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ABSTRACT 


Combat system reliability is central to creating combat power, determining 
logistics supportability requirements, and determining systems’ total ownership costs, yet 
the Marine Corps typically monitors only operational availability. While acceptable 
operational availability may be achieved through intensive maintenance and the stocking 
of needed repair parts in large quantities, this increases the logistics burden on the combat 
commander and is costly in terms of personnel, time, and funding. 

Data required to compare system reliability requirements in source documents, 
such as the Operational Requirements Document and the acquisition contract, to achieved 
reliability of fielded systems is generally not collected, maintained, or available. 
Contractual obligations to attain system reliability, if any, could not be enforced, and any 
increase in sustainability costs associated with unmet reliability thresholds is borne by the 
Marine Corps, draining scarce funding from other priorities. 

This research interprets data and perspectives, as collected from a reliability 
management survey administered to acquisition workforce professionals, and collectively 
summarizes common inhibitors of effective reliability management, why they occur, 
lessons learned, and potential methods for mitigating the inherent risks. The results 
ascertain a variety of technical, programmatic, managerial, incentive, and procedural 
issues that the Marine Corps encounters concerning system reliability requirements and 


achievement. 
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I. INTRODUCTION 


A. BACKGROUND DISCUS SION 

In today’s environment of aging weapon systems, there is an increased need for 
Operations & Support (O&S) funding. However, because of DoD budgetary constraints, 
there has been a trend in recent years to utilize discretionary modernization funds in an 
effort to fund shortfalls in O&S accounts. As a result, DoD’s current «quisition 
approach is to acquire products and services to meet military needs that will provide the 
best value to the government over the life cycle of the product or service. Consequently, 
performance parameters, to include reliability achievement, must be considered in 
relation to Total Ownership Costs (TOC) vice simply considering the initial procurement 


costs of weapon systems. 


Acquisition decisions made early in the life cycle of weapon systems can have a 
tremendous impact on the availability and sustainment of Marine Corps equipment. 
Thus, highly reliable systems are extremely important as they serve as force effectiveness 
multipliers that significantly contribute towards increased system availability, a reduced 
logistical footprint, and a net reduction in total ownership costs, which equate to 
increased funds for modernization. Therefore, it is imperative that a primary goal of 
systems acquisitions is to field reliable equipment that is both capable and supportable 


from the start. 


Both the inherent reliability designed into weapon systems and the estimates of 
such reliability have significant impacts on weapon system readiness and cost for decades 
as the reliability estimates provide the basis for initial life-cycle supportability decisions, 
including integrated logistical support packages. Specifically, such estimates contribute 
to determining the initial procurement of spare parts and support equipment, concept of 
logistical support, the number and training of mechanics, readiness estimates, ope ration 
and support costs, and Program Objectives Memorandum (POM) planning. Therefore, 
the effect of low inherent reliability, as well as the effect of under - or over-estimating the 
reliability of weapon systems, will cause already limited dollars to be allocated unwisely 


as unanticipated life cycle costs accumulate and cause an additional need for O&S dollars 


in later years. Consequently, it is imperative to obtain, verify, and utilize accurate 
reliability forecasts early in the life cycle process and to attempt to tie contractors to 


readiness and LCC thresholds through reliability estimates. 


Fortunately, there are many early opportunities for addressing reliability within 
weapon systems acquisitions. Initially, the Requirements Generation Process can serve 
as a primary tool for the Marine Corps to document quantifiable system reliability 
requirements in the Operational Requirements Document (ORD) in the form of Key 
Performance Parameters (KPP). Additionally, the reliability requirements can be used in 
source selection as we convert specific performance specifications into contractual terms, 
which could perhaps include an inherent reliability goal. From here, the Systems 
Engineering Process allows the contractor to build to such required performance 
specifications. Additionally, once contractors submit their reliability estimates, program 
planning and organizational management can emphasize an independent and rigorous 
reliability testing process throughout the development phase in order to demonstrate the 
required reliability performance levels to ensure the system will operate in the field as 
intended. Lastly, while not an upfront opportunity, comparison and assessment of 
achieved field reliability to contractor reliability estimates could be conducted throughout 


weapon system maturation to ensure attainment of system reliability as planned. 


However, due to procedural, management, and incentive issues, the Marine Corps 
is faced with inhibitors to effective reliability management, and thus, the acquisit ion 
community has not been able to fully take advantage of such reliability management 
opportunities. Ultimately, as a result, the warfighter is not provided with a reliable and 
supportable weapon system that is capable of being sustained within its life cycle cost 
threshold. 

B. OBJECTIVES AND PURPOSE OF THE RESEARCH 

The purpose of this research is to evaluate how weapon system reliability 
performance is managed throughout the acquisition process by identifying common 
inhibitors and enablers of effective reliability management, why they occur, lessons 
learned, and potential methods for mitigating the inherent risks. The results of the thesis 
are intended to directly benefit Program Managers while providing insight into the 


improved sustainability of future systems. Understanding that reliability estimates 
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provide the basis for initial life cycle supportability decisions, the acquisition community 


must utilize effective procedures, as well as develop management strategies and 


techniques to address reliability risks. The research ascertains procedural issues that the 


Marine Corps deals with concerning reliability requirements in the acquisition process as 


well as common management and incentive issues faced by program management 


offices. The resulting analysis includes conclusions and recommendations applicable to 


the acquisition community. 


RESEARCH QUESTIONS 


C. 


D. 


1. 


Primary Research Question 


What strategies should be used to better manage weapon system reliability 
during the life cycle of major weapon systems? 


Subsidiary Research Questions 
How does reliability affect Life Cycle Cost and Operational Availability? 


What are the existing policies, regulations, and guidance that govern 
reliability of weapon systems available to the Combat Developer, Program 
Management Office, and Contractor? Do they provide adequate 
guidance? 


How does the Marine Corps address reliability performance of weapon 
systems during the Requirements Generation Process? 


How can the Marine Corps create and adhere to a contractual oblig ation in 
the form of quantitative system reliability requirements that forces 
contractors to consider reliability equally with other system parameters 

such cost, schedule, and performance? 


How is system reliability addressed during developmental and oper ational 
testing, and is the Marine Corps adequately testing to determine and 
demonstrate the required reliability performance levels? 


Is there a significant difference between contractors’ reliability estimates 
and achieved reliability of fielded systems as obtained from Marine Corps 
logistics systems, and if so, is the Marine Corps adequately assessing the 
data during the maturation of weapon systems in order to alleviate future 
contractor reliability inaccuracies? 


Is the Marine Corps maintenance rate, in the form of MTBM, a feasible 
surrogate for comparison with traditional failure rate, in the form of 
MTBF, as obtained from contractors? 


SCOPE, LIMITATIONS, AND ASSUMPTIONS 


The research for this thesis was completed in collaboration with a similar 


concurrent study conducted by Studies and Analysis Division, Marine Corps Combat 
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Development Command (MCCDC) under the sponsorship of Marine Corps Systems 
Command (MARCORSYSCOM), entitled “Sustainment Consequences of Acquisition 


” 


Decisions.” The thesis assesses pre-fielding programmatic and technical decisions that 
influence reliability of fielded systems (MARCORSYSCOM study plan “Sustainment 
Consequences .. .”). Specifically, the scope of this research includes an evaluation of 
reliability management within the Marine Corps acquisitions process from numerous 
perspectives to include: 1) a review of the relationship between reliability, operational 
availability, logistics support, and life-cycle costs, 2) a review and assessment of current 
DoD and Marine Corps policy, guidance, and regulations regarding reliability, 3) an 
examination of reliability requirements documentation and its relevance in source 
selection, 4) an assessment of transforming ORD reliability performance specifications 
into contractual obligations, 5) an evaluation of the extent to which reliability 
requirements are being demonstrated during testing, 6) a comparison and assessment of 
reliability requirements and contractor reliability estimates to actual achieved reliability 
of fielded systems, and 7) an analysis of the Marine Corps’ adequacy of comparing and 
assessing the aforementioned data during the maturation of weapon systems. The 
research will aide in assessing the accuracy and completeness of reliability estimates for 
fielded systems while identifying techniques to improve the accuracy of reliability 
estimates during systems development. Furthermore, a comparison of documented 
reliability requirements and pre-fielding estimates to achieved reliability will provide 
beneficial insight into achieving future readiness (MARCORSYSCOM Draft SOW 


“Sustainment Consequences .. .”). 


The data collected is limited to mature Critical/Pacing items included in the 
Quarterly Readiness Reports to Congress (M1A1 tank, AAV family of vehicles, LAV 
family of vehicles, Ston truck family of vehicles, HMMWV family of vehicles, LVS 
family of vehicles, and the M198 Howitzer). The analysis is limited to an assessment of 
reliability management issues, while not specifically addressing technology driven 
reliability problems. While the research is limited to selected principle end items, it is 
assumed that the challenges, issues, and potential solutions can be applied to other end 


items in the Marine Corps acquisition process. 


E. METHODOLOGY 


In an effort to determine the current environment for reliability management 


within Marine Corps acquisitions, the researcher administered an electronic survey 


(Appendix B) to various personnel within the Program Offices of specific critical/ pacing 


end items. The questions pos ed were intended to emphasize the perspective of program 


management leadership on the varied tasks involved with reliability management 


(Masiello, p. 4). Specifically, the survey was intended to conduct an examination of 


current policy and regulations, reliability requirements documentation, contractual 


obligations, developmental and operational test data, and readiness/maintenance data. 


The survey utilized was a modification of a previously designed reliability performance 


survey intended to gather data within a specific Army Program Executive Office in 


pursuit of similar research objectives (Ryan, pp. 91-97). 


The methodology used in this thesis research consisted of the following steps: 


Through a review of existing publications, examine and document the 
relationship between reliability, logistics, life-cycle support costs, and 
readiness 


Review and examine the adequacy of current DoD and Marine Corps 
policy, guidance, and regulations that govern reliability 


Conduct a review of the acquisition process, from determining needs 
requirement through sustainment operations and support 


Through the combination of data collection from the Fleet and reliability 
survey responses from the acquisition community: 


Determine the extent to which the Marine Corps organizations 
involved throughout the acquisition process consider reliability 


Determine how the Marine Corps addresses reliability performance 
in the requirements generation phase 


Review the current process and methods of transforming ORD 
requirements into quantifiable contractual obligations 


Determine the extent to which reliability requirements are 
demonstrated during testing 


Determine if contractor reliability estimates are retained, and 
determine the achieved reliability data of mature fielded systems 


Compare and assess the predetermined reliability requirements and 
contractor estimates to achieved reliability of mature systems. 
Determine and evaluate the Marine Corps’ adequacy at conducting 
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the same comparison throughout the maturation of weapon 
systems. 


e Assess the collected data to identify policy, managerial, and 
procedural issues involved with current reliability management in 
the acquisition process 


e Recommend policy and procedural changes to reliability 
management throughout the acquisition process and provide 
insight into the improved sustainability of future systems through 
the obtainment of accurate reliability estimates from contractors 


F. ORGANIZATION OF THE RESEARCH 


This thesis contains six chapters. 


Chapter I introduces the subject of reliability as a basis for the study while 


providing the objectives, scope, methodology, organization, and benefits of the research. 


Chapter II provides a background and overview of reliability while defining 
reliability and related concepts. The relationship between reliability, logistics, life-cycle 
support costs, and operational availability will be addressed. Additionally, this chapter 


discusses the tools and techniques available for reliability analysis. 


Chapter III is a brief overview of the acquisition process from the Requirements 
Generation Process through Sustainment Operations and Support. Additionally, this 
chapter discusses the participants and organizations involved in the process. Also, the 
current DOD and Marine Corps policies, regulations, and guidanc e that establish the 
basis within which the acquisition community should operate to manage reliability within 


a program will be discussed. 


Chapter IV provides the program demographics and background of the systems 
that are a part of this study and presents the aggregate results of the data collection from 
the reliability survey. This data indicates how the respective programs have implemented 


reliability management processes and highlights significant examples and experiences. 


Chapter V analyzes and compiles the key issues and challenges associated with 
reliability to include issues with existing policy and guidance on reliability, reliability 
requirements determination and documentation, contracting for reliability, developmental 
and operational testing, and comparison and assessment of reliability requirements and 


estimates to achieved reliability. 


The final chapter makes conclusions and recommendations, provides answers to 
the primary and secondary research questions, and recommends areas for further 
research. 

G. BENEFITS OF THE RESEARCH 

According to Marine Corps Systems Command, there are currently no known 
studies within the Marine Corps comparing the relationship of reliability, availability, and 
maintainability (RAM) to Operational Availability and determining its impact on Future 
Readiness thresholds (MARCORSYSCOM Draft SOW “Sustainment Consequences . . 
.”, p. 3). Thus, the primary benefit of this study is the identification of policy and 
program management issues with respect to weapon system relia bility and providing 
recommendations for areas of potential improvement. The research is intended to 
directly benefit the acquisition community by identifying common potential inhibitors, 
identifying their underlying root causes, providing lessons learned, and suggesting 
methods for managing and reducing inherent risks associated with achieving reliability 
performance requirements. Additionally, attaining accurate contractor reliability 
estimates, used as a basis for initial life cycle supportability issues, will benefit the 
Marine Corps by optimizing the use of constrained resources and improving the 
operational force materiel readiness posture (MARCORSYSCOM Draft SOW 


“Sustainment Consequences . . .”). 


THIS PAGE INTENTIONALLY LEFT BLANK 


Il. RELIABILITY OVER VIEW AND BACKGROUND 


When in a fight to the death, one wants to employ all one’s weapons to the 
utmost. I must say that to die with one’s sword still sheathed is most 
regrettable. - Miyamoto Musashi, Book of Five Rings 


A. INTRODUCTION 

The purpose of this chapter is to provide a fundamental understanding of 
reliability and its importance within weapon systems acquisitions. This will be 
accomplished by addressing the relationship between reliability, logistics, life-cycle 
support costs, and operational availability. However, an overview of reliability and 
related concepts will first be required to provide a common frame of reference and 
establish a general basis of understanding for subsequent discussions. Accordingly, this 
chapter also discusses the alignment of process ownership between the Program 
Management/Weapon System Management (PM/WSM) and Supply Chain Management 
(SCM) organizational elements while detailing the changes recently implemented within 
the Marine Corps to best accommodate life cycle management of its equipment. Lastly, 
tools available for reliability analysis will be briefly introduced. 


B. RELIABILITY DEFINED: RELATED DEFINITIONS, CONCEPTS, AND 
MEASURES 


In order to address the role of reliability in the logistics community, it is 
imperative to understand the terms and definitions most widely associated with defining 
and discussing reliability. The intent of this section is to provide basic quantitative and 
qualitative knowledge of reliability-related definitions and concepts required to plan for, 
design, produce, and implement an effective and efficient logistic support capability. Of 
particular emphasis within weapon systems acquisitions are the qualitative measures of 
reliability and logistics, which must be addressed in order to ensure logistics 
requirements are adequately specified, evaluated, and modified for improvement. In 
addition to reliability itself, other measurements are utilized to characterize the reliability 


of a system and its components. 


1. Reliability 


The probability that a system or product will perform in a satisfactory 
manner for a given period of time when used under specified operating 
conditions (Blanchard, p. 25). 


e When considering component reliability, the term “system” can be 
extended to include components or subsystems that can be considered as 
an entity 

e The term “satisfactory” indicates that specific criteria must be established 


to determine what satisfactory operation/service is 


e For a hardware item to be reliable it must do more than meet an initial 
factory performance requirement — it must operate for a given period of 
time in the actual application for whic h it is intended. “Time” represents a 
measure against which the degree of system performance can be related. 


Inherent reliability is the potential reliability of a system (inherent as 
designed), assuming an ideal operating and support environment. 


As evident from the preceding clarifications, the concept of reliability is often 
utilized without precise definition, while the terminology is non-standard throughout the 
logistics community and tends to depend on the Service and/or system. In the broadest 
sense, reliability is associated with dependability, with successful operations, and with 
the absence of breakdowns or failures (Lewis, p. 1). However, while creating DoD 
requirements documentation and contract specifications, it is very important that all main 
concepts are addressed in an unambiguous way so that all parties involved understand the 
terms. Furthermore, to adequately conduct engineering analyses, reliability must be 
defined quantitatively as a probability. Thus, one must consider the time parameter in 
order to assess the probability of completing a given function as scheduled. The 


reliability function, R(t), may be expressed as: 
R(t) = Pr(T >= |" f@dr=1-FO (2.1) 
Let T be a random variable that represents the time until the next failure, f(t) be 
the probability density function, and F(t) be the cumulative density function of T. 


Then the reliability function, R(t), is defined as the probability that the failure will 


not occur until time f. 
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Assuming that the time to failure is described by an exponential density function, 


the reliability function, R(t), is: 


R(t) =Pr(T >= | ” f (dt =1-F() = | “he “dx =e™ (2.2) 


where t is the time period of interest, and e is the natural logarithm base (2.7183), and A 
is the instantaneous failure rate (Blanchard, p. 37). It is important to note that the 
reliability function as depicted above is in terms of an exponential distribution. This 

means that the unit’s failure rate is constant over the period 4, the reliability for a new 
mission is independent of the age of the unit and is a function of its failure rate and the 
duration of the new mission only. This is commonly used in many applications under the 
presumption that all like components are being utilized in the exact same manner with the 
same stresses imposed upon them. In reality, the failure characteristics of different 

components vary considerably depending upon their usage. Other applicable dens ity 
functions include the normal, binomial, exponential, Poisson, gamma, and Weibull 

distributions (Kececioglu, p. 202). However, explanation of such distributions are 
beyond the scope of this thesis. 

2. Failure Rate 


The number of item failures of per measure of unit life, where failure is 
defined as the termination of an item’s ability to perform a required 
function (Hoyland and Rausand, p. 10). 


The failure rate is expressed as: 
numberof failures 1 


fee zt 
totaloperatinghours MTBF 





(2.3) 
When determining overall failure rate, it is important to address all system factors 


that cause the system to be inoperative at a time when satisfactory system operation is 


required. A combined failure rate is presented in Table 2.1. 
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Assumed Factor 
Consideration (instances/hour) 


(a) Inherent reliability failure rate .000392 
(b) Manufacturing defects .000002 
(c) Wear-out rate -000000 
(d) Dependent failure rate .000072 
(e) Operator-induced failure rate -000003 
(f) Maintenance-induced failure rate .000012 


(g) Equipment damage rate .000005 


Total combined factor .000486 





Table 2.1. Combined Failure Rates. (From: Blanchard, p. 40) 


3. Mean Time Between Failure (MTBF) 

For a particular interval, the total functional life of a population of an item 

divided by the total number of failures with the population (DSMC, 

“Acquisition Logistics Guide”, p. 10-2). 

MTBE serves as the basic technical measure of reliability, and thus, the measure 
becomes a key element in support planning. In simplified terms, MTBF is the average 
time between required corrective (unscheduled) maintenance actions. MTBF should not 
be used interchangeably with failure rate, and in fact, MTBF is the inverse of the failure 


rate: 


MTBF =— (2.4) 


It is important to distinguish why MTBF needs to be calculated for equipm ent. 
The calculation of this time is necessary in order to determine whether the mean time 
between failures is increasing, decreasing, or remaining constant with age. As equipment 
ages, its MTBF decreases until the cost of keeping that item operational is more than the 


cost of buying a new item. Estimates of when maintenance costs will exceed acquisition 
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costs are questionable without mean time between failure calculation (Enholm, p. 1). In 
other words, MTBF data analysis can help to determine if equipment is in the “wear-out” 
phase of its life cycle and at the end of its economic useful life. 

4. Mean Time Between Maintenance (MTBM) 

MTBM includes both preventive (scheduled) and corrective (unscheduled) 
maintenance requirements. It includes consideration of reliability MTBF and MTBR. 
MTBM may also be considered as a reliability parameter and can be expressed as: 


1 Pa 
1 i‘ 1 + fpt 
MTBM MTBM 


unscheduled 


MTBM = (2.5) 


scheduled 


where fpt (=1/MTBM ;) is the frequency of the preventive maintenance actions per system 
operating hour, or the preventive maintenance rate. Also, MTBM unscheduled (SaMe as 
MTBF) is the mean interval of unschedu led maintenance and MTBM geheautea 18 the mean 


interval of scheduled maintenance (NPS Logistics Engineering principle). 


It should be obvious that MTBM is not the same measurement as MTBF due to 
the inclusion of preventive maintenance actions. However, the Marine Corps is often 
forced to substitute MTBF with MTBM due to lack of operational usage data needed to 
calculate MTBF. The feasibility of this substitution will be discussed in more thorough 
detail later in the thesis. 

5. Availability 

The probability that an item (system) is in an operable and committable 

state at the start of a mission when the mission is called for at a random 

point in time. “Is the equipment available in a working condition when it 

is needed?” (DSMC, “Acquisition Logistics Guide”, p. 10-3) 

Availability is frequently used as a measure of system readiness, and thus, the 
user is often most concerned about this parameter. There are numerous expressions of 
availability, all of which are based on the standard mathematical relationship between 
“up time”, “down time”, and “total time.” In other words, over long operating periods, 
availability can essentially be expressed as a relationship between uptime (reliability) and 


downtime (DSMC, “Designing Quality into .. .”, p. B-1). 
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a. Inherent Availability (Aj) 

Inherent availability takes into account only items of systems design. 
Additionally, it assumes an ideal support environment and includes only active corrective 
maintenance time in calculation of downtime while excluding preventive maint enance 
time and servicing times as well as supply, administrative and personnel delays. Inherent 
availability is expressed in terms of its designed mean time between failures (MTBF) and 
its designed mean time to repair (or active repair time) (MTTR) given that it has failed: 


MTBF MTBF 


ee ee (2.6) 
(MTBF +MTTR) (MTBF +M,,) 


A,= 


where M_, = mean corrective maintenance time. 

b. Achieved Availability (A,) 

Achieved availability is calculated when preventive maintenance is 
included in the relationship. However, an ideal (no delay) support system is still 
assumed, which excludes Logistics Delay Time (LDT) and Administrative Delay Time 
(ADT): 

MTBM 


ee (2.7) 
MTBM +M 


a 


where M = mean active maintenance time (both preventive and corrective maintenance 
activities) and MTBM is the mean time between maintenance, both corrective and 
preventive. 

Cc. Operational Availability (A,) 

Operational Availability is a function of the reliability and maintainability 
of the equipment and is a commonly used measure of weapon system readiness. It is the 
most desirable form of availability to be used in helping assess a system’s potential under 
fielded conditions whereas achieved availability and inherent availability are primarily 
the concern of the developing organization in its interface with the contractor (DSMC, 
“Acquisition Logistics Guide”, p. 10-4). Specifically, operational availability is the 
probability that a system, when used under stated conditions in an actual operational 
environment, will operate satisfactorily when called upon at any random time. 
Additionally, operational availability includes all of the sources of non-operable time, 


active and inactive to include supply and administrative delay times, corrective and 
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preventive maintenance, and personnel/maintenance technician delays. The value 
provides both the percentage of time that a system is in a mission capable status in the 
long-run and the percentage of weapon systems in mission capable status: 


MTBM _Numberof MissionCapableSystems 


=—________= 2.8) 
MTBM + MDT Totalnumberof systems 


oO 


where MDT = maintenance downtime, or the total elapsed time required to repair and 
restore a system to full operating status. Maintenance downtime (MDT) includes mean 
active maintenance (M), logistics delay time (LDT), and administrative delay time 


(ADT). 


Despite which expression of availability used, it is obvious that reliability 
is a major driver in the numerator of these relationships. 
6. Reliability Component Relationships 
Overall system reliability is a function of the reliability of subsystems and 
components. With today’s technology, systems performance may often be increased at 
the expense of increased complexity; the complexity usually being measured by the 
number of required components and parts. However, unless compensating measures are 
taken to improve the reliability of the components, system reliability will decrease. This 
is because if nothing else is changed, reliability decreases with each added component. 
In such cases of increased system complexity, reliability can only be maintained if 
component reliability is increased or if component redundancy is built into the system. 


However, each of these solutions, in turn, must be measured against incurred costs 


(Lewis, p. 3). 


The decrease in reliability due to increased system complexity may be expressed 
in terms of the product rule. The reliability of the system is the product of reliabilities of 
the individual subcomponents. In other words, if the component failures are mutually 
independent in a series form, the reliability of the system with N nonredundant 
components is: 


R=RR,...R,..Ry (2.9) 


As depicted, in a series network, all components must operate in a satisfactory 
manner if the system is to function properly. Connecting subsystems in a series tends to 
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decrease reliability, since the reliability of the entire system is equal to the product of the 


individual reliabilities of that system. 


However, from a reliability perspective, system components can be integrated in 
parallel form, enabling system developers to increase system reliability through increased 
redundancy in the system. Ina parallel network, a number of the same components are in 
parallel, and thus, all components must fail in order to cause total system failure. For a 
system with n identical components, the reliability expression for the system is: 


R=1-—(1-R)" (2.10) 


Parallel redundant networks are used primarily to improve system reliability 
(Blanchard, p. 45). Additionally, various levels of reliability can be achieved through the 
application of combining series and parallel networks. In fact, a combination of both 
types of systems is commonplace and almost unavoidable. Once systems engineers 
determine the reliability of individual components, overall system reliability can be 
empirically calculated. Ultimately, the true source of system reliability rests with the 
performance of individual components and subsystems (Chaudhary, p. 26). 

7. Reliability Bathtub Curve 

The reliability of a system and its components will fluctuate throughout their 
development, production life cycle, and operational usage. Additionally, product updates, 
system changes or modifications, and maintenance actions further affect the reliability of 
systems and their components. However, assuming a negative exponential distribution, 
the failure rate is relatively constant during the mature stages of a system life cycle as 
shown in Figure 2.1. It is during this relatively stable portion of the curve that the 
exponential failure law applies. However, when systems are initially operational, there 
are usually a higher number of failures mostly attributable to poor manufacturing 
techniques, poor quality control, poor workmanship, insufficient burn-in or break-in, 
improper installation, insufficient debugging, human error, and other causes. As a result, 
the initial failure rate is higher than anticipated before leveling off to the constant failure - 
rate region. Likewise, when a system reaches a certain age, it enters its wear -out life 
period where the failure rate once again increases (Kececioglu , p. 74). It should be noted 


that the curve would vary depending upon the type of system and its operational usage. 
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Figure 2.1. Reliability Bathtub Curve. (From: Kececioglu, p. 74) 


Effective reliability programs require the assessment of reliability at key decision 
points along the growth curve. Data availability for making projections obviously 
increases as the program and its tests progress. For example, during the early life period, 
known as the infant mortality period, reliability estimates must be made on information 
obtained from stress calculations, proven component data from similar equipment, 
accelerated testing, and potential problem analysis, all of which are reliability analysis 


tools to be discussed later in this Chapter. 


Ultimately, the actual reliability level of a system and its components, as well as 
the confidence in the estimated level, increases with the test program and its 
corresponding corrective actions. Attempts must be made to obtain the require d times -to- 
failure and success-and-failure data in an effort to prepare a reliability bathtub curve, 
plotting the failure rate of a system versus its age. Such a curve enables the estimation of 
(a) the optimum break-in testing period and burn-in time, (b) the optimum warranty time 
and its cost, (c) the optimum preventive replacement time, and (d) the spares 


requirements (Kececioglu, “Reliability Engineering Handbook’, p. 37). 
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Cc. RELATIONSHIP BETW EEN RELIABILITY, LOGISTICS, LIFE CYCLE 
SUPPORT COSTS AND READINESS 


Early materiel life cycle decisions during the acquisition process have a 
significant impact on future operational availability and life cycle cost of weapon 
systems. This is largely due to the fact that reliability characteristics that are inherent 
within the system design actually dictate the requirements for the subsequent 
maintenance and support of that system throughout its life cycle (Blanchard, p. 252). In 
addition to actual inherent reliability associated with system design, under- or over- 
estimations of the reliability of weapon systems in development dramatically and often 
adversely affect life cycle cost and operational availability as the reliability estimate 


provides the basis for initial life cycle supportability decisions. 


Weapon systems must be designed to be supportable for the warfighter, capable 
of being maintained effectively and efficiently throughout their planned life cycles, 
ultimately enabling the warfighter to focus his efforts on his primary task of winning 
battles and providing him with equipment capable of doing so. Therefore, the DOD must 
remain focused on the goal of providing systems that maximize their operational 
availability (A,) within the allocated life-cycle cost (LCC) of the program. When 
considering readiness and supportability objectives within budgetary constraints, system 
reliability emerges as the prominent life cycle cost and readiness driver for defense 
weapons systems. Thus, it is critical to consider the role of reliability in planning for 
integrated logistical support in the early stages of planning and design as well as 
throughout the entire acquisition process. However, before attempting to specify 
quantitative reliability requirements and considering managerial or procedural methods to 
improve reliability, one must be able to clearly establish the link between reliability, life 
cycle cost, and readiness. 

1. Impact of Reliability on Operational Availability 

The ability to successfully complete a mission is directly dependent on the 
weapon performing that mission without experiencing a mission critical failure. In other 
words, weapon system reliability directly affects the ability of the Marine Corps to 
perform its mission. With this in mind, it becomes clear that “reliability isn’t everythin g, 


it is the only thing” (Eaton Email, 25 April 2001). 
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The following formula indicates that there is a definite direct relationship between 


reliability, maintainability, and readiness (A,): 





UPTIME 
ee 
bi az uptime - MTBM is OT +ST (2.11) 
uptime +dowtime MTBM+MDT OT+ST+ALDT+CMT+PMT 
UPTIME DOWNTIME 


where, 
OT = Operating Time 
ST = Standby Time 
ALDT = Administrative and Logistics Down Time 
CMT = Corrective Maintenance Time 
PMT = Preventive Maintenance Time 


As “uptime” or Mean Time Between Maintenance (MTBM) increases as a result 
of increased reliability, operational availability (or readiness) also increases (DSMC, 
“Program Managers Tool Kit’, p. 43). 

2. Impact of Reliability on Life Cycle Costs 

While equipment failure due to poor reliability can be catastrophic, leading to life 
or death implications, reliability of many products may be viewed primarily in economic 
terms. Much of the projected life-cycle cost for a given system can be greatly impacted 
by decisions made during the early stages of advanced planning and conceptual and 
preliminary design. Management and design decisions at this point can have a major 
impact on the activities and operations in all subsequent phases of the life cycle. Thus, it 
is critical to consider reliability and its affect on logistical support in the early stages of 
planning and design in an effort to avoid unplanned excessive O&S costs throughout a 
system’s life cycle and not postpone reliability considerations to a downstream activity. 
The need to look beyond short-term initial cost of procurement and acquisition and 
address system life-cycle cost is obvious, and experience has shown that logistics 
requirements can have a major impact on overall life-cycle cost (Blanchard, p. 4). 
Understanding that initial life cycle supportability requirements to include integrated 


logistics support is based on reliability estimates, it becomes clear that reliability needs to 
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be recognized as a significant factor throughout the life cycle while assuming a major 
role in research, design, production, and system performance during operational use. An 
increased focus on reliability can lead to reduced life cycle support costs, equating to 
increased funds available for recapitalization and modernization of forces. Likewise, 
because of its recognized importance, it is mandatory for all program managers with the 
Department of Defense to plan for and execute measures to ensure their program 


accounts for the user’s RAM objectives (DoD 5000.2-R). 


Along with the latest revision to the DoD 5000 series acquisition directives in 
October 2000, the Secretary of Defense issued a memorandum that outlined six major 
themes in the updated documents. One of the major themes is that, “The acquisition 
process must consider both performance requirements and fiscal constraints. 
Accordingly, cost must also be an independent variable in programmatic decisions.” The 
theme, known as, Cost As an Independent Variable (CAIV), is an initiative intended to 
put focus on life-cycle costs by considering both performance requirements and fiscal 
constraints by making cost and performance trade offs. Over the past decade, the relative 
importance of LCC has greatly increased, and it is now mandatory for the major 
acquisition category programs. Additionally, many contemporary political issues dictate 
that the control of costs associated with procurement and life cycle management of 
weapon systems receive an unprecedented level of management attention (DSMC, 


“Acquisition Logistics Guide”, p. 12-1). 


The concept of CAIV must by utilized in establishing an effective acquisition 
strategy. Per DoD 5000.2-R, the acquisition strategy shall address methodologies “to 
acquire and operate affordable DoD systems by setting aggressive, achievable cost 
objectives and managing achievement of these objectives”. A strategy that considers the 
total cost to the government over the entire cradle -to-grave cycle of the system is 
“necessary to provide balance and perspective to the program in consideration of the 
performance and schedule requirements to avoid suboptimization”. In this regard, 
program managers primary focus should be on minimizing life cycle cost (DSMC, 


“Acquisition Strategy Guide”, p. 2-12). 
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a. Background and Components of LCC 

DOD TOC is comprised of costs to research, develop, acquire, own 
operate, and dispose of weapon and support systems, other equipment and real property, 
the costs to recruit, retain, separate and otherwise support military and civil ian personnel, 
and all other costs of business operations of the DOD. Defense Systems TOC is defined 
as Life Cycle Cost. LCC (per DoD 5000.4M) includes not only acquisition program 
direct costs, but also the indirect costs attributable to the acquisition program (i.e., costs 
that would not occur if the program did not exist). For example, indirect costs would 
include the infrastructure that plans, manages, and executes a program over its full life 


and common support items and systems. 


For purposes of cost estimating, LCC is typically divided into research and 
development (R&D), procurement, operations and support (O&S), and disposal. Life 
Cycle Costs involves all costs associated with the system life cycle, to include the 


following: 


° Research and development (R & D) cost. Those costs incurred from 
program initiation at the conceptual through the end of engineering and 
manufacturing development. R&D costs include the cost for feasibility 
studies, modeling, tradeoff analyses, engineering design, development, 
fabrication, assembly and test of prototype hardware and software, system 
test and evaluation, associated peculiar support equipment, and 
documentation. 


e Procurement cost. Includes the costs associated with producing or 
procuring the prime hardware, support equipment, training, data, initial 
spares, and facilities. 


° Operation and support (O&S) cost. Consists of all costs incurred by the 
DOD to field/deploy the system including personnel, consumable and 
reparable parts, fuel, shipping, and maintenance. Includes the cost of 
sustaining operation, personnel and maintenance support, spare/repair 
parts and related inventories, test and support equipment maintenance, 
transportation and handling, facilities, modifications and technical data 
changes, and so on. 


e System retirement or disposal cost. Captures costs associated with 
deactivating or disposing of a materiel system at the end of its useful life. 
(DSMC, “Acquisition Logistics Guide”, pp. 12-3 — 12-4). 
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As depicted by the categories listed above, life cycle cost of a weapon 
system begins with the determination of a mission requirement and continues through 
design, development, production, operation, support, and eventually the disposal and 
demilitarization of the system at the end of its useful life. It is widely accepted within the 
acquisition community, that the costs of operating and supporting a weapon system far 
exceed the actual procurement costs incurred through the design, development, and 
production of a new system. Although the percentage of life-cycle costs attributable to 
each element is not identical for all weapon systems, there is little variation across the 
range of various systems. The historical life-cycle cost percentage breakdown for major 
defense weapon systems is depicted in Figure 2.2 (OSD CAIG, “O&S Cost Estimating 
Guide”). 
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Figure 2.2. Weapon System Life Cycle Cost Breakdown. 


While production may be viewed as the most costly portion of the 
program per unit of time, it actually only amounts to roughly 30% of the LCC. Based on 
these figures, it becomes readily apparent that the largest cost driver in the life of a 
system is the O&S phase. To further compound this figure, when today’s aging systems 
exceed their originally intended life expectancy, O&S costs can actually form 75 -90% of 


a system’s LCC (Parker, p. 275). Understanding that an increasing portion of the defense 
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budget is being consumed by growing O&S expenditures, there has understandably been 
considerable effort to reduce such costs. Ultimately, the increase of fun ds available for 
recapitalization and modernization of legacy systems will result through the reduction of 


O&S funds. 


In the past, total system cost has either not been obvious or has been 
somewhat ignored due to incentive and managerial issues, partic ularly those costs 
associated with operation and support. As previously discussed, a major portion of the 
projected life-cycle cost for a given system or product results from the consequences of 
decisions made during the early phases of program planning ad system conceptual 
design. Referring back to Figure 2.2, while the greatest proportion of life cycle costs 
occur during the operation and support phase of a program, the greatest opportunity for 
influencing these costs occurs during the early phases of the program as shown in Figure 
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Figure 2.3. | Commitment of Life-Cycle Cost. (From: Blanchard, p. 82) 


The recent CAIV acquisition reform initiative is a way of developing life- 
cycle cost targets for the system to be acquired and constraining the system design trade - 


offs by the target cost of system ownership. Prior to the CAIV concept, the Design -to- 
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Cost approach (DTC) was very prominent within the acquisition community. However, 
the DTC approach had primarily concentrated on controlling system procurement costs, 
rather than life-cycle cost. As a result, DTC created the wrong incentives for former 
program management offices, resulting in programs that did not adequately address 
sustainment and life cycle cost consequences of early acquisition decisions. 

b. Life Cycle Cost Analysis 

Life cycle cost analysis is typically part of the supportability analysis, 
discussed later, and is conducted to address the total cost of a system and its supporting 
activities throughout its planned life cycle. Such an analysis includes the estimation of 
the system life cycle cost (design and development, production and/or construction, 
system utilization, maintenance and support, and retirement/disposal costs), high-cost 
contributors, cause-and-effect relationships, potential areas of risk, and identification of 
areas for improvement or cost reduction (Blanchard, p. 176). Due to the fact that much 
of the downstream cost is the consequence of design and management decisions made 
during the early stages of conceptual and preliminary design, the use of life cycle cost 
analysis is critical if a program management office is to assess whether or not the system 
can be operated and supported in an effective and efficient manner throughout its 


intended life cycle. 


Many factors are involved with the estimation of life cycle costs. 
Specifically, reliability considerations, estimates, and the accuracy of such estimates play 
a significant role in LCC estimations. The fundamental objective of LCC reduction 
analysis is to identify the cost drivers that most significantly affect life cycle costs. Such 
analyses allow for trade off considerations with respect to different courses of action. 
During each phase of the acquisition cycle, engineers and managers provide prompt 
feedback regarding the costs of new or alternative designs or other economical solutions 
with respect to their effect on LCC forecasts. Likewise, engineers and managers must 
achieve a proper balance between acquisition decisions and costs and the resulting 
(predicted) operation and support costs. Figure 2.4 illustrates the design linkage with 
operation and support cost drivers (DSMC, “Designing Quality into Defense Systems”, p. 
41-42). 
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Figure 2.4. Design and Life Cycle Cost Linkage. (From: DSMC, “Designing Quality 
into Defense Systems”, p. 42) 


There are countless examples of how reliability improvements in both 
Government and industry have resulted in substantial cost reductions. It is well know 
throughout the current acquisition community that initial investments in the design, 
development, and production of reliable weapon systems can have significant impacts on 
reducing O&S costs and ultimately LCC. Such an example is the DoD’s Minuteman I 
missile system which implemented a reliability improvement study that eventually led to 
a 30% reduction in the failure rate. The cost-effectiveness analysis revealed a return of 
eight dollars for every dollar invested in reliability improvement. The net savings over a 
ten-year period was expected to be $160,000,000 (Kececioglu, p. 23). Another example 
of the potential cost savings can be found with the F-105 weapon system, which, by way 
of implementing a reliability improvement program, increased system reliability from 
.7263 to .8986. While the reliability program nonrecurring costs were estimated at 
$25,500,000, the annual savings in maintenance costs were estimated at $54,000,000 


(Kececioglu, p. 24). It is clear that while upfront investments in reliability may increase 
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initial procurement costs, the significant savings resulting from the potential reductions in 
O&S and LCC quickly outweigh any upfront costs. 

c. Break Even Analysis 

A program must consider cost during reliability and maintainability design 
balancing activities. Fortunately, Life Cycle Cost models are available and often used as 
a vehicle by which estimates for operation, performance, reliability and maintainability, 
and cost are traded off to obtain “design to” target goals which collectively represent a 
balanced design. For the purpose of considering cost trade-offs, additional relationships 
are developed which define how cost changes as reliability and maintainability is varied 
from a baseline. Specifically, as a system is made more reliable, the operating cost 
should decrease since there are fewer failures to repair. At the same time, it is anticipated 
that acquisition cost (development and production) will increase to attain higher 


reliability in the system (DSMC, “Designing Quality into...”, p. 11). 


As discussed, improvements in system reliability, to a feasible extent, 
dramatically decrease system LCC. However, increasing system reliability beyond 
feasible technological levels may require an enormous amount of resources to be 
consumed during research and development (R&D) to the point that the cost savings 
from improved reliability may not offset such costs, resulting in less than optimal LCC. 


The theoretical relationship between system reliability and LCC is depicted in Fig ure 2.5. 


LIFE-CYCLE COST 


OPERATION & 
SUPPORT COST 


RELIABILITY ——————> 





Figure 2.5. Reliability and LCC Tradeoff. 
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For proper economic analysis, one must consider the costs associated with 
the entire life cycle of a system, evaluating the trade-off between increased early 
investments in teliability improvement and the resulting future cost savings. When 
comparing alternatives, a program management office must consider both the aspects of 
cost effectiveness and the point in time where one alternative becomes more cost - 
effective than another alternative. A break-even analysis is an approach where the 
cumulative costs for two or more investment alternatives (or programs) are estimated, 
projected, and compared with respect to time. In the event that the break -even point is 
realistic in terms of expected system life, then it may cost-effective to consider the 
increased early investment during Research and Development phases in order to achieve 
higher system reliability. Figure 2.6 provides a comparison of two alternatives where it 
appears that the increased investment during R&D results in a more cost-effective option 


in the long run. 
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Figure 2.6. | Break-Even Analysis. (From: Blanchard, p. 89) 


d. Cost Effectiveness 
It is important to understand that when decidin g upon the optimal level of 


reliability to be designed, manufactured, and maintained into a product, it is not 
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necessarily the point at which the cost to own, operate, and maintain the product for its 
desired life is minimum. Rather, the primary objective should be to develop a system 
that is most cost-effective, within the constraints of operational and maintenance 
requirements. In other words, the acquisition community should not aim to strictly 
minimize LCC, and instead, should consider cost effectiveness as it relates to the measure 
of a system in terms of mission fulfillment (system effectiveness) and total life cycle 
costs (Blanchard, p. 34). Cost effectiveness involves a cost-benefit analysis factor 


employed for decision-making purposes. 


When cons idering cost effectiveness, the aspects of effectiveness must be 
quantified and depend upon the specific mission or system characteristic that a program 
desires to specify and measure. While measuring effectiveness, one must consider: 


e System performance ad physical parameters: capacity, delivery rate, 
power output, range, accuracy, volume, speed, weight, etc. 


e System operational and support factors: availability, dependability, 
capability, operational readiness, reliability, maintainability, etc. 


e Total life-cycle cost: research and development, production/construction 
cost, operation and maintenance cost, retirement and disposal cost 
(Blanchard, p. 83) 


In order to achieve a desirable cost effectiveness, a relationships must be 
established between performance and operational parameters and cost. Figure 2.7 
illustrates an example of the relationship between reliability (MTBF) and total life cycle 
cost, where the objective is to design a design a weapon system that meets a specified 
reliability level within a given budget level and yet be most cost-effective. System 
design characteristics are evaluated in terms of reliability and cost, and as a result, design 
changes are recommended in an effort to achieve the point on the curve near the 


minimum cost (Blanchard, p. 88). 
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Figure 2.7. Reliability versus Cost. (From: Blanchard, p. 88) 


There is a significant increase in costs associated with achieving higher 
levels of reliability. In fact, the marginal increase in reliability bec omes increasingly 
smaller and the marginal cost becomes increasingly larger as developers attempt to 
maximize the level of reliability. In other words, in may be relatively inexpensive to 
increase reliability from 50% to 70% while it may be far more costly to increase system 
reliability from 98% to 99%. Therefore, it is not typically optimal to strive for 100% 
reliability. Figure 2.8 illustrates the diminishing marginal gain associated with achieving 


higher levels of reliability. 
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Figure 2.8. Reliability S-Curve. (NPS Logistics Engineering Class Notes) 


e. Life Cycle Cost Models 

Numerous commercial life cycle cost models have been developed in an 
effort to help Program Managers structure and analyze large amounts of data used to 
support major LCC decisions. One of the major advantages of the LCC models is their 
ability to provide early input to the front-end design analysis stage of the Concurrent 
Engineering (CE) and Logistic Supportability (LS) processes. Basically, the models 
available are database managers that have the capability, to varying degrees, to import, 
modify, analyze, integrate, and manage large amounts of data from many different 
sources. Reports can be generated that display or project the overall effects and results of 
program decisions on existing or alternative system designs, including risks thereof while 
storing a baseline of program decisions. The life cycle cost models provide a design and 
support system tradeoff with sensitivity and comparative analyses, providing the 
flexibility of rapidly assessing the reliability, LCC and logistic supportability impacts of 
various equipment configurations and other design supportability options (Sterling, 
“Analysis of LCC Models for DoD”). Some of the life cycle cost models available to 
Program Management Offices include but are not limited to EDCAS, ACEIT, FLEX+, 
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CASA, and the COCOMO model, all of which offer varying degrees of advantages as 
well as disadvantages relative to the others. The specific application of the models will 
not be discussed in this text as it is beyond the scope of the thesis. 
D. SUPPORTABILITY ANALYSIS 

Supportability analyses are a wide range of related analyses that should be 


conducted within the systems engineering process. Specifically, supportability analysis 
(SA) is 


. .. an iterative analytical process by which the logistic support necessary 

for a new (or modified) system is identified and evaluated. The SA 

constitutes the application of selected quantitative methods to (1) aid in 

the initial determination and establishment of supportability criteria as an 

input to design; (2) aid in the evaluation of various design alternatives; 

(3) aid in the identification, provisioning, and procurement of the various 

elements of maintenance and support; and (4) aid in the final assessment 

of the system support infrastructure throughout the utilization phase 

(Blanchard, p. 24). 

Reliability characteristics inherent within the system design actually dictate the 
requirements for the subsequent maintenance and support of that system throughout its 
lifecycle, and thus, program offices must establish the appropriate logistic support 
requirements in the early stages of conceptual design (Blanchard, p. 252). However, in 
addition to actual inherent reliability associated with system design, under- or over- 
estimations of the reliability of weapon systems in development can dramatically, and 
often adversely, affect life cycle cost and operational availability as the reliability 
estimate provides the basis for initial life cycle supportability decisions. Therefore, 


accurate reliability predictions and thorough analyses are required as an integral input to 


the supportability analysis. 


The supportability analysis includes two major areas of focus. The firs t is the 
accomplishment of design trade-off studies, level of repair analyses, life-cycle cost 
analyses, and related activities directed toward the objective of designing for 
supportability. The second area of focus involves the evaluations of the system design 
configuration, as it exists at the time, with the objective of defining logistic support 
resource requirements (i.e., spare/repair parts, test and support equipment, number of 


maintenance personnel, level of personnel training, etc.). With the identification of 
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specific logistics requirements identified, the provisioning, procurement, and acquisition 
process commences (Blanchard, p. 355). Ultimately, the supportability analysis leads to 
a database that assists in identifying the specific requirements leading to the development 
of the maintenance and support infrastructure. The overall intent is to design or develop 
a system that will meet the specified operational requirements in an effective and 
efficient manner by maximizing system effectiveness while minimizing life cycle cost. 
E. RELIABILITY ANALYSIS AND AVAILABLE TOOLS 

The reliability analyses can be used to define the quantitative parameters for a 
system, subsystem, or component, and it is often expressed in number of failures in a 
given set period of time, set number of cycles, or a set number of operations (i.e., rounds 
fired, number of starts, etc). As engineering data become available, reliability prediction 
serves as a check on the design in relation to the system requirement, indicating areas of 
incompatibility that may need evaluated for design improvement. As _ previously 
discussed, the level of reliability achieved in fielded systems directly affects operational 
availability and sustainment requirements. Therefore, successful system de signs require 
that component and system reliability be predictable. This requires that a reliability 
program be established to assess the reliability of system components. Accurate data is 
crucial in establishing reliability information, and the more data available, the greater the 


confidence that can be expressed in the estimated or predicted reliability level. 


During logistical support planning, the Marine Corps is forced to rely upon 
estimates, and unfortunately, reliability data is often difficult to obtain, as it is acquired 
through observing the failure of products and their components. This requires life 
testing, in which a number of items are tested until a significant number of failures occur. 
However, such tests are often very expensive, since they are destructive, and to obtain 
meaningful statistics, substantial numbers of the system or subsystem must fail. The tests 
are also time consuming, since few unbiased acceleration methods are available to greatly 
compress the time to failure, the test time may be comparable or longer than the normal 
product life. Reliability data is also collected from field failures once a product is put 
into operational use. However, this is a lagging indicator and is not nearly as useful as 
results obtained earlier in the development process (Lewis, p. 49). Additionally, it is 
important that reliability be considered in the concept and design process because 
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identifying and correcting related problems in later stages of the life cycle has an adverse 


cost leverage as shown in Figure 2.9. 
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Figure 2.9. _ Relative Costs of Problem Correction versus Program Phase. (DSMC, 
“Designing Quality into . . .” p. 30) 


Multiple potential opportunities are present throughout the acquisition life cycle 
to address reliability. Beginning with the initial requirements generation, through each 
iteration of the systems engineering process, and ultimately during post-production, 
reliability must be planned for, monitored, accessed, and improved during the matu ration 
of a weapon system (Ryan, p. 1). The program’s application of special reliability tasks 
enhances the capability of satisfying the warfighter or user’s needs. However, reliability 
tasks must be fully integrated into the total technical program and _ be performed 
concurrent with other engineering tasks to ensure reliability is designed -in before design 
maturity reaches a stage when engineering changes become costly to implement 
(Reliability, Maintainability, and Supportability Guidebook, SAE Internatio nal, p. 70). 
While the list of key reliability tasks, below, serve slightly different purposes, they are 
applicable to varying equipment types, and range in depth, scope, and complexity of the 


task, if properly conducted, all, in some capacity, can provide a valuable contribution to 
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the design and development of a system with respect to reliability performance. These 


tasks must be tailored to fit the particular program need. Furthermore, the tasks listed are 


only those with the widest acceptance and application within program management and 


are not all-inclusive. 


Reliability Requirements Definition 

Reliability Program Plan 

Reliability Design Standards/Guides/Checklists 
Environmental Criteria 

Reliability Modeling 

Reliability Allocation and Apportionment 

Reliability Prediction 

Subcontractor/Supplier Monitoring and Control 
Reliability Design Evaluation 

Failure Modes Effects and Criticality Analysis FMECAS) 
Process Failure Modes and Effects Analysis (PFMEA) 
Reliability Development/Growth Test (RD/GT) 
Weibull Analysis 

Failure Reporting, Analysis, and Corrective Action (FRACAS) 
Software Reliability Assessment 

Parts Control Program 

Environmental Stress Screening (ESS) 

Reliability Qualification Test (RQT) Program 
Probabilistic Design Assessment for Reliability 

Fault Tree Analysis 

Part Stress Derating 

Worst Case Circuit Analysis 


The integrated analyses can include any number of tools, practices, or techniques 


to realize reliability and supportability characteristics. The tasks above, or some 


combination of them, should be selectively applied to each program based on the 


program’s life cycle, system complexity and type, technology advancement, and schedule 


and cost constraints. If the selected reliability tasks are appropriately tailored for scope 
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and depth and adequately integrated with other program tasks, an effective reliability 
program will result ((Reliability, Maintainability, and Supportability Guidebook, SAE 
International, p. 71). 


F. PROGRAM MANAGER/WEAPON SYSTEM MANAGEMENT 
IMPLEMENTATION 


Program Management/Weapon Systems Management (PM/WSM) is defined as 
“the planning, organizing, acquisition, controlling, sustainment, and disposal of weapon 
systems and secondary items in support of validated Marine Corps requirements,” while 
Supply Chain Management is defined as “the planning, organizing, and controlling of 
supply chain activities for the Marine Crops wholesale and retail supply business to 
maintain and support assigned principle end items and secondary items” (PM/WSM 
“Activities Definitions”). Under the recent PM/WSM initiative, traditional roles, 
responsibilities, resources, and billets of Marine Corps Systems Command 
(MARCORSYSCOM) and Marine Corps Logistics Base (MARCORLOGBASES) were 
realigned to optimize Life Cycle Management of weapon systems. The initiative was 


established to “‘clearly delineate authority, responsibility, and accountability of managers 
and organizations” (Williams, PM/WSM Slide Show dtd 17 Jan 01). 


Prior to the Program Manager/Weapon System Manager Implementation efforts, a 
major weapon system was procured and fielded at MARCORSYSCOM and was passed 
on to MARCORLOGBASES for Sutainment/Life Cycle Management. As a result of this 
disjointed process, major weapon systems entered the Fleet and encountered severe 
readiness and supportability problems (MARCORSYSCOM Study Plan “Sustainment 
Consequences .. .”, p. 1). It has been argued that prior to the implementation of 
PM/WSM, the incentives in place for program managers caused them to focus on short - 
term program objectives that they were evaluated on such as procurement cost, schedule, 
and performance. Additionally, few if any, incentives were in place that encouraged 
program mangers to analyze long-term sustainment and life cycle cost consequences of 
their early acquisition decisions. However, under the realignment of responsibilities 
within Materiel Command (MATCOM), LOGBASES, and MARCORSYSCOM, Marine 
Corps Systems Command became responsible for the availability of equipment through 


the entire materiel life cycle. As a result, the decisions made early in the life cycle of a 
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system, that often have a tremendous impact on availability and sustainment, will directly 
impact the program management offices throughout the life span of the respective 
weapon systems. 
G. CHAPTER SUMMARY 

This chapter established the definite relationships between reliability, logistics, 
life-cycle support costs, and operational availability. In doing so, the researcher 
illustrated the fact that the reliability of a weapon system directly impacts the operational 
availability and the life cycle cost of the system, making it of fundamental importance to 
PM, logisticians, and warfighters alike. Appropriately, the core of logistical support 
planning focuses on reliability, in an attempt to ensure that warfighters are provided with 
capable, supportable, and cost effective weapon systems that enable them to successfully 


complete the mission on the battlefield. 


Chapter III will provide an overview of the acquisition process while providing 
specific reference to opportunities within systems’ life cycles for program managers to 


address reliability. 
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Il. BACKGROUND AND OVERVIEW OF RELIABILITY 
WITHIN THE ACQUISITION PROCESS 


Reducing the cost to acquire and operate the department’s equipment 
while maintaining a high level of performance for the user is my highest 
priority. - Under Secretary of Defense for Acquisition and Technology 
memorandum dated 04 December 1995 


A. INTRODUCTION 

One of the first major steps in the development of reliability focus in DoD 
acquisitions came in July 1980, when the DoD indicated an emphasis on reliability and 
maintainability by publishing a policy directive on the subject in the form of DoDD 
5000.40. Until recently, there has been a lack of management emphasis on the support 
engineering disciplines such as reliability, and thus, the timely application of engineering 
techniques had not always been practiced. As a result, the efforts were not as supportable 
and cost effective as they could have been. Today, with the high level of TOC interest in 
the DoD, the management attention and interest is present, and as a result, we continue to 
make advancements in the way of reliability-focused acquisitions (Reliability, 


Maintainability, and Supportability Guidebook, SAE, p. 64). 


This chapter provides the reader with background information on the defense 
acquisition process and serves to establish an understanding of general opportunities for 
reliability management within the process. First, an examination of current DoD and 
Marine Corps policies, regulations, and guidance is provided to establish the basis within 
which the acquisition community must operate to manage reliability within a program. 
Next, an overview of the acquisition process is provided, highlighting opportunities for 
reliability management throughout the process. Finally, the chapter will conclude by 
examining the existing roles, metrics, and incentives that guide the various organizations 
and individuals involved in the acquisition process. 


B. DOD AND MARINE CORPS POLICIES, REGULATIONS, AND 
GUIDANCE ON RELIABILITY 


Past and present Administrations and Congresses have instituted many initiatives 
to improve the acquisition of defense systems. In particular, the publication of the DoD 


5000 Series of Directives in February 1991 resulted from the culmination of a 
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cooperative effort within the DoD to streamline policy by standardizing acquisition 
procedures. As a result, all acquisition tasks that were common among the service 
components were combined into the top-level policy, resulting in the cancellation of 65 
other directives. The Department of the Navy implemented the DoD directives in 
SECNAVINST 5000.2A in December 1992, resulting in the cancellation of 39 additional 
directives. The Marine Corps implementation of the SECNAV policy in May 1994 
resulted in the cancellation of 14 additional policy directives. “The resulting product of 
these three efforts is a single policy source outlining broad acquisition procedures for 
Marine Corps acquisition programs” (USMC PM Acquisition Procedures Handbook, p. 
1-1) 


Additionally, 1996 was a noteworthy year for acquisition policy changes. 
Defense policies now included acquisition streamlining, integrated product development, 
performance specifications, and the prohibition of most military specifications and 
standards. The 15 March 1996 reissuance of DoDD 5000.1 and DoD 5000.2-R (ater 
with change 1 of 13 December 1996) promulgated these policy changes in directive 
format. The major focus of the new policies are teamwork (IPTs), teamwork with 
industry, tailoring empowerment, only performing value-adding tasks, employing Cost 
As an independent Variable (CAIV), a preference for commercial items, and use of best 


practices (DSMC, “‘Acquisition Logistics Guide”, p. 1-2). 


There are many sources of reliability guidance to assist program offices with 
achieving reliability requirements, a few of which are mandatory while others are 
discretionary or even cancelled. In fact, upon searching the Defense Acquisition 
Deskbook website for DoD (discretionary or mandatory) documents containing the word 
“reliability,” 213 documents were located. Such policy, regulations, and guidance have 
been established to emphasize the importance of reliability and to ensure that the 
acquisition community is striving toward improved reliability techniques. As previously 
mentioned, much of the guidance is very broad scoped, providing little detail as to 
specific reliability actions to be taken in the acquisition process. Additionally, the 
amount of mandatory guidance is minimal and has further decreased in recent years due 
to acquisition reform initiatives. This section serves to provide a general overview of 


some of the sources of guidance as well as the nature of the guidance. 
38 


1. Mandatory Guidance 

DoD 5000.2-R, Mandatory Procedures for Major Defense Acquisition Programs, 
states that as part of the acquisition strategy for a given program, program Managers shall 
develop and document a support strategy for life-cycle sustainment and continuous 
improvement of product affordability, reliability, and supportability, while sustaining 
readiness. RAM activities addressed in DoD 5000.2-R are summarized below: 

e The PM shall establish RAM activities early in the acquisition cycle 


e The PM shall develop RAM system requirements based on the 
Operational Requirements Document (ORD) and Total Ownership Cost 
(TOC) considerations, and then state them in quantifiable, operational 
terms that are measurable during development and operational testing 


e Reliability requirements shall address mission reliability and logistics 
reliability 

° Availability requirements shall address the readiness of the system 

e Maintainability requirements shall address servicing, preventive, and 


corrective maintenance 


e The PM shall plan and execute RAM design, manufacturing de velopment, 
and test activities so that the system elements, including software, used to 
demonstrate system performance before the production decision reflect the 
mature design (DoD 5000.2-R) 


DoD 5000.1 is another source of mandatory guidance which directs that: 

Acquisition program managers shall focus on logistics considerations 
early in the design process to ensure that they deliver reliable systems that 
can be cost-effectively support and provide users with the necessary 


support infrastructure to meet peacetime and wartime readiness 
requirements (DoD 5000.1) 


Lastly, SECNAVINST 5000.2B, Section 4.3.6 — Reliability, Maintainability, and 
Availability — serves to interleave the higher-level policy. 

2. Discretionary Guidance 

In addition to the limited mandatory guidance on reliability, there is an abundance 
of discretionary guidance, consisting mostly of Military Handbooks. Such discretionary 
guidance most typically emphasizes integration of reliability into the design, 
manufacturing, and support process while providing recommended tools and procedures 


for doing so. It is important to note that because the handbooks serve as guidance only, 
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they cannot be cited as requirements. Due to amount of existing documents, only the 


most relevant sources will be identified in this section. 


Military Handbook (MIL-HDBK)-781A, Handbook for Reliability Test Methods, 
Plans, and Environments for Engineering, Development Qualification, and Production , 
provides a list of reliability test methods, reliability test plans, and envir onmental profile 
data that can be used as a guide when testing systems for contractual reliability 


requirements during developmental testing. 


MIL-HDBK-189, Reliability Growth Modeling, outlines reliability growth 
concepts and methodologies for management of reliability growth during the 
developmental stage by presenting fundamental concepts followed by details for concept 


implementation. 


MIL-HDBK-502, Acquisition Logistics, offers guidance on acquisition logistics as 
an integral part of the systems engineering process, to include technical and management 
activities associated with the design, development, test, production, fielding, sustainment, 
and improvement modifications. Additionally, the handbook offers methods to “identify, 
consider, and trade-off support considerations with other system cost, schedule, and 
performance elements to arrive at an optimum balance of system requirements that meet 


the user’s operational and readiness requirements” (MIL-HDBK-502, Section 4). 


The “US Marine Corps Program Managers Acquisition Procedures Handbook” 
implements DoD, DON, and Marine Corps directives on Defense Systems Acquisition. 
Additionally, the handbook serves as a “summation of Marine Corps and, if appropriate, 
MARCORSYSCOM philosophy and policy regarding selected acquisition subject areas” 
(USMC PM Acquisition Procedures Handbook, p. ii). However, the handbook offers 
minimal guidance concerning reliability related actions to be taken during the respective 


phases of the acquisition process. 


Lastly, the DoD Defense Acquisition University (DAU) has published a series of 
guidebooks that are utilized during their courses of acquisitions instruction at Fort 
Belvoir, Virginia. While designed to be technical management educational guides 
written from a DoD perspective, the guidebooks reflect the latest DoD acquisition 


policies and procedures as described in the 5000 series. 
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DoD- and Marine Corps-specific policy, regulation, and guidance on reliability 
exist to establish the basis within which the acquisition comm unity should operate to 
manage reliability within a program. While there is an abundance of DOD 
documentation concerning reliability within the acquisition process, most is discretionary 
with little mandatory guidance and procedures on the subject. Additionally, what is in 
print is often very vague in nature and provides little specific guidance to the Program 
Mangers. 

C. OVERVIEW OF THE ACQUISITION PROCESS 

The Program Manager must consider reliability and other acquisition logistics 
management activities throughout the system development to ensure the design and 
acquisition of cost-effective, supportable systems and to ensure that these systems are 
provided to the warfighter with the necessary support infrastructure for achieving the 
user’s peacetime and wartime readiness requirements (DSMC, “Acquisition Logistics 
Guide”, p. 3-11). Consequently, logistics requirements must be initially planned from the 
beginning, and subsequently into the system design process. Reliability tasks must be 
fully integrated into the program and be performed concurrent with other engineering 
tasks to insure reliability is designed-in before design maturity reaches a stage when 
changes become costly to implement. In the past, the emphasis on delivering capability 
(performance) in a timely manner (schedule) within procurement cost objectives has 
often overridden reliability and total ownership cost considerations. Likewise, logistics 
has been considered as a “bill to be paid later,” and thus, DoD often struggles to 
efficiently and effectively maintain its existing mature weapon systems on today’s 


battlefields. 


In the defense sector, there has been a recent emphasis on early logistical planning 
during the acquisition process that has evolved through the concept of integrated logistic 


support (ILC), defined as a: 


Disciplined, unified, and iterative approach to the management and 
technical activities necessary to (1) integrate support considerations into 
system and equipment design; (2) develop support requirements that are 
related consistently to readiness objectives, to design, and to each other; 
(3) acquire the required support; and (4) provide the required support 
during the operational phase at minimum cost. (DSMC, “Integrated 
Logistics Support Guide’) 
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As a result of the recent focus on post deployment logistical supportability, there 
has been an increased emphasis on the early opportunities for addressing reliability 
within weapon systems acquisition. Initially, the Requirements Generation Process can 
serve as a primary tool for the Marine Corps to document quantifiable system reliability 
requirements in the Operational Requirements Document (ORD) in the form of Key 
Performance Parameters (KPP). The reliability requirements can be used in source 
selection as DoD converts specific performance specifications into contractual terms, 
which should perhaps include an inherent reliability goal. The Systems Engineering 
Process allows the contractor to build to required reliability performance specifications. 
Once contractors submit their reliability estimates, program planning and organizational 
management can emphasize an independent and rigorous reliability testing process 
throughout the development phase in order to demonstrate the required reliability 
performance levels to ensure the system will operate in the field as intended. While not 
an upfront opportunity, comparison and assessment of achieved field reliability to 
contractor reliability estimates could be conducted throughout weapon system maturation 


to ensure attainment of system reliability as planned. 


The subsequent sections will provide an overview of the participants involved in 
the acquisition process, a summary of the process itself, and the opportunities to address 
reliability throughout the process. 


1. Organizations and Participants in the Marine Corps Acquisition 
Process 


Weapons systems acquisition is a very complex process, involving many different 
participants at varying levels. This section, a summation taken from the “USMC 
Program Managers Acquisition Procedures Handbook,” provides an overview of the 
organizations and participants involved as well as a brief summary of their respective 


roles. 


Before proceeding, it is important to note that the organizational chain of 
command is not the same as the systems acquisition chain and that certain levels are 
responsible for requirements while others are responsible for implementing those 


requirements. 


42 


The chain of authority for Marine Corps systems acquisition starts at the 
Department of Defense level where the responsibility for acquisition policy and major 
program decision authority has been placed with the Under Secretary of Defense, 
Acquisition, Technology, and Logistics (USD, (AT&L)). The position of USD (AT&L) 
is subordinate only to the Secretary and Deputy Secretary of Defense. In the systems 
acquisition hierarchy, the USD (AT&L) is the Defense Acquisition Executive (DAE), 
acting as the ultimate program decision authority on certain major programs preparing to 


move from one Milestone to the next. 


Immediately below the USD (AT&L) in the systems acquisition hierarchy is the 
position of the Assistant Secretary of the Navy, Research, Development, and Acquisition 
(ASN, RDA). The ASN (RDA) performs the same role for the Secretary of the Navy that 
the USD (AT &L) does for the Secretary of Defense. ASN (RDA) is the sole decision 
authority within the Department of the Navy (DoN) for major Navy/Marine Corps 
programs, and is responsible for Navy acquisition policy. ASN(RDA) also serves as the 
Component Acquisition Executive (CAE) for the Navy, and is referred to as the Navy 
Acquisition Executive (NAE). 


The next position in the Marine Corps acquisition hierarchy is the Commandant 
of the Marine Corps (CMC). The CMC is responsible for determining requirements and 
ensuring the resources (funding and people) for those requirements. However, the CMC 
is not directly involved in the program decision process. Instead, the CMC appoints a 
Milestone Decision Authority (MDA) to act in his behalf in the acquisition 
decision/policy process, similar to the roles performed by the NAE and USD (AT&L). 
The Commander, Marine Corps Systems Command (COMMARCORSYSCOM) 
performs the MDA role for the Marine Corps. Before proceeding, we must distinguish 
between Marine Corps Systems Command and the Marine Corps Combat Development 


Command (MCCDC). 


There are two major functions involved in systems acquisition — requirements 
determination and acquisition. As previously discussed, the CMC’s role at the top level 
is primarily with requirements determination. However, the Commanding General, 


MCCDC acts as the CMC’s agent in this process. Part of MCCDC’s overall mission is to 
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translate deficiencies and desired capabilities into operational requirements. Meanwhile, 
the mission of MARCORSYSCOM, simply stated, is to take a validated requirement and 
turn it into reality, in the form of warfighting weapon systems and equipment. The CG, 
MCCDC acts as the Commandant’s agent in developing requirements while the 
Commander, MARCORSYSCOM acts as his agent in acquiring the systems that fulfill 
those requirements. Clear boundaries between requirements determination (CG, 
MCCDC) and acquisition (COMMARCORSYSCOM) exist to effectively translate 


operational needs into stable and affordable acquisition programs. 


The Program Managers (PMs) are responsible for directing the efforts of 
acquiring the systems to fulfill the validated requirements. They are responsible for 
taking the requirement from concept to an operational system. According to the “USMC 
PM Acquisition Procedures Handbook,” in broad terms, the Program Managers have 
three major responsibilities: “Cost, Schedule, and Performance.” It should be noted that 
the handbook mentions “logistical supportability” as a part of performance criteria for 
which program managers are responsible while indicating that Integrated Logistics 
Support is the process by which to achieve such criteria (USMC PM Acquisition 
Procedures Handbook, Chapter 1). 


With the inclusion of the PM, we have completed the streamlined program 
decision relationship in the acquisition hierarchy: PM to CMDR, MARCORSYSCOM, 
to CAE (NAB), to DAE. Figure 3.1 generically depicts the Marine Corps participants in 
the acquisition process, from generation of the requirement and program initiation, to 


fielding and post-deployment support. 
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Marine Forces, Atlantic 
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Test & Evaluation Activity 6. Operational Test & 
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System Fielded 


Deputy Chief of Staff, 
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Headquarters, USMC 7. Post-Deployment Support 


Marine Corps Logistics 
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Figure 3.1. Participants in the Acquisition Process. (From: USMC PM Acquisition 
Procedures Hnbk, p. 1-14) 
2. Acquisition Phases and Milestones 
Along with the recent changes to the DoD Directive 5000 series, a new DoD 
Systems Acquisition Process model was created which was intended to deliver advance 
technology to the warfighters faster, reduce total ownership costs and improve 
affordability, and deploy interoperable and supportable systems. Some professionals may 
argue that there is little significant difference between the old and new models depicted in 
Figure 3.2 aside from the stages and milestones renamed. However, others point out that 
the new model integrates testing and evaluation throughout the system; allows for 
“evolutionary developments” based on time-phased (ORD) requirements; offers multiple 
process paths or entry points into the process depending on conceptual and technical 


maturity of the existing system; separates technology development from system 
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integration; ensures “entrance criteria” before entering the next phase which serves as a 
gate for the Milestone Decision Authority to decide if the program should continue; 
includes operation, support, and disposal as part of the acquisition process; and requires 
full funding at system development vice program definition, creating more competition 
between competitors. Despite which model a program is guided by, DoD controls the 
acquisition process through a series of tailorable Milestones and Phases that serve as 
decision points and goals to be achieved. Additionally, phases help focus the effort and 
define the necessary activities for effective management. However, due to the dynamic 
nature of DoD acquisitions, Program Management must remain flexible (NPS MN3331 


Class Notes, “Principles of Systems Acquisition and Program Management’). 


Concept Program Definition Engineering & ’ Production, 
Exploration & Risk Reduction Manufacturing Fielding/Deployment 
Hs Development (EMD) 


Previous Model 


! Cc 


Concept Component System System Low-Rate Full-Rate 
Exploration Advanced Integration Demo Initial Production 


Development Production & Deploy 
’) Review Review 


Review 
Risk Reduction & 


Demonstration Production & Deployment 


Concept & Tech Development 
<— Pre-System Acquisition ~>4————————_ System Acquisition —————————* Sustainment 4— 


New 5000 series Model 





Figure 3.2. | The System Acquisition Models. 


Throughout the remainder of this thesis, future references of the phases and 
milestones most often cite the previous model due to the fact that the systems examined 
in this study were procured under such processes. 

a. Acquisition Program Baseline (APB) 
Each program has an APB that defines the cost, schedule, perf ormance, 


and supportability measures that it must meet, with thresholds and objectives defined that 
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serve as boundary parameters within which the PMs operate. The APB serves as a 
“contract” of sorts between the PM and the MDA. Reliability related parameters such as 
MTBF, A,, and MTBM exist for each program either in the Performance or 
Supportability sections of the APB. The acquisition program baseline status of each 
program is reviewed once a quarter and at major reviews (Ryan, p. 36). 

b. Acquisition Decision Memorandum (ADM) 

When a program reaches a major milestone or experiences a significant 
change in its program parameters, the outcome is documented in an ADM. The ADM 
serves to document decisions made by the MDA, and typically includes additional 
directive statements that the PM must comply with. The Acquisition Decision 
Memorandum statements and directives are an opportunity for the MDA to encourage the 
achievement or improvement of reliability levels, while placing exit criteria, constraints, 
or follow-on actions related to reliability on the programs. 

3. Requirements Generation Process 

As this section will indicate, the Requirements Generation Process can serve as an 
initial primary tool for the Marine Corps to document quantifiable system reli ability 
requirements in the Operational Requirements Document (ORD) in the form of Key 
Performance Parameters (KPP). Reliability requirements definition is the translation of 
warfighters’ operational requirements into specific reliability requirements that can be 
defined, designed to, and measured. The requirements definitions are incorporated in 
written specifications that contain numerical statements of required reliability and precise 
description of the performance and environmental requirements that must be met to 
achieve the numerical reliability requirements. Close attention must be given to such 
reliability requirements because they are eventually used as contractual and acquisition 
devices to assure mission success and performance over time (Reliability, 
Maintainability, and Supportability Guidebook, SAE, p. 73). 

a. Mission Needs Statement (MNS) 

All acquisition programs are based on identified, documented, and 
validated mission needs, resulting from ongoing assessments of current and projected 
capability with respect to changing military threats and the National Security Strategy 
(NSS). Within the Marine Corps, part of MCCDC’s overall mission is to translate 
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deficiencies and desired capabilities into operational requirements. Requirements 
determination and revision follow an established process, beginning with the Capability 
Review System within MCCDC where deficiencies are manifested by the Fleet Marine 
Force (FMF) through Fleet Operational Need Statements (FONS), the Marine Corps 
Lessons Learned System (MCLLS), Mission Area Analysis (MAA), and the Marine 
Corps Master Plan (MCMP). Additionally, the natural expiration of service life of 
equipment is factored into the process. A material solution to a deficiency begins with a 
broad statement of the requirement as outlined in a Mission Need Statement (MNS), 
developed by MCCDC, and sent to the Assistant Commandant of the Marine Corps 
ACMC for approval. The MNS is a non-system specific statement of operational 
capability need written in broad operatio nal terms. It is non-specific by design and offers 
a materiel solution in one of three ways: improvements to an existing system, 
procurement of a non-developmental item, or begin a new research and development 
program. Subsequent approval and signature of the MNS by the ACMC constitutes a 
“validated requirement” and initiates Milestone A. Following the Mission Need 
Statement, MCCDC performs individual Analysis of Alternatives (AoA), and although 
not a requirements document, it forms the basis for an Operational Requirements 
Document (ORD), which is also drafted by MCCDC (USMC PM Acquisition Procedures 
Handbook, p. 1-6). It is through the AoA that an approach is formulated to set and refine 
life cycle cost objectives. 

b. Operational Requirements Document 

The ORD is a key document in the acquisition process, for it translates the 
MNS into more detailed and refined performance capabilities and characteristics of a 
proposed concept or system. To do so, the ORD defines the requirement, states the 
numbers of systems and where they should be fielded, and describes the specific 
operational capabilities required. MCCDC acts as the Combat Developer, and develops 
the Operational Requirements Document, which details the required system capabilities 
and characteristics to include the user’s definition of system reliability parameters in 
operational terms. MCCDC is ultimately responsible for defining the requirements 
relative to the reliability of the system. It is at this stage that defining the “essential 


qualitative and quantitative readiness and logistics supportability requirements in 
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operational concepts and requirements documents is the most effective way for users to 
influence the design of their systems” (Department of Air Force, Instruction 10-602, 
1994). Typically, this is defined in terms of operational availability and mission duration 
needs. As directed in DoD 5000.2-R, these operational performance parameters are to be 
stated as Objectives and Thresholds. Section 2.6 of DoD 5000.2-R states 
supportability factors are integral elements of program performance 
specifications. However, support requirements are not to be stated as 
distinct logistics elements, but instead as performance requirements that 


relate to a system’s operational effectiveness, oper ational suitability, and 
life-cycle cost (DoD 5000.2-R). 


Reliability, along with cost, schedule, and performance, should act as 
equal partners in the requirements generation process. An effective way to ensure that a 
system maximizes its operational availability is to include robust reliability goals in the 


ORD. 


At each milestone, beginning with program initiation, thresholds and 
objectives initially expressed as measures of effectiveness (MOEs) and minimum 
acceptable requirements for the proposed concept or system are documented by the user 
or the user’s representative in the ORD to quantify system level performance. Thresholds 
and objectives in the ORD consider the results of the analysis of alternatives and the 
impact of affordability constraints (DSMC, “Acquisition Logistics Guide”, p. 5-2). The 
Combat Developer’s definition of the intended reliability requirement is an essential 
element in establishing the basis for any successful reliability program. Whether the 
requirements result from the needs of the user or from internal goals identified by a 
design or project organization, well-defined requirements are needed. Conversely, poorly 
defined requirements lead to conflicts in direction and inefficiencies in the application of 
engineering and management resources. If the requirements are defined properly, close 
adherence to the ORD is necessary for a successful logistics program (Reliability, 


Maintainability, and Supportability Guidebook, SAE, p. 42). 


Reliability requirements determination is not accomplished in a vacuum. 
In fact, developing quantitative operational reliability requirements, like all other ORD 


requirements, is a collaborative process between the combat developer (MCCDC) and the 
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materiel developer (MARCORSYSCOM) using Integrated Product Teams (IPTs). This 
process provides a balanced solution between the best estimate of what is required to 
meet the warfighter’s effectiveness, suitability, and survivability needs, and that which is 
actually affordable and technically achievable within program funding, risk, and time 
constraints (Ryan, p. 13). 

Cc. Key Performance Parameters 

While the ORD serves to establish minimum acceptable operational values 
for broad performance parameters, the Marine Corps has the opportunity to include 
quantifiable and understandable reliability requirements as Key Performance Parameters 
(KPPs) in the ORD. A KPP is a capability or characteristic that is so significant that 
failure to meet the threshold can be cause for the concept or system selection to be 
reevaluated or the program to be reassessed or terminated. By placing reliability 
requirements as KPPs in the ORD, contractors would be required to test to reliability. 
Such KPPs would likely ensure adequate logistics weight in source selection. 
Unfortunately, reliability (as well as availability and maintainability) requirements are 
usually not KPPs, and when there are cost or schedule overruns, reliability is sacrificed. 
In reality, reliability KPPs should be expressed with both threshold (minimally accep ted 
values) and objectives (what the user desires and what the PM is attempting to obtain). 
Then, given a system’s reliability goal that is clearly defined by the Combat Developer as 
a KPP in the ORD, the designer understands what reliability the system should be 


“designed to.” 


Part of the intent of new 5000 series and the new acquisition model is to 
reduce Total Ownership Costs (TOC) by minimizing the number of mission -oriented Key 
Performance Parameters. Upper levels of DoD believe that this maximizes the PM’s and 
contractor’s flexibility to make cost/performance tradeoffs without the unnecessary 
higher-level permission, proving to be essential to achieving cost objectives. Therefore, 
the number of threshold items in requirements documents and acquisition program 
baselines are strictly limited. The threshold values represent true minimums, and the 
requirements should be stated in terms of capabilities rather than technical solutions and 
specifications. While reliability related KPPs typically are not n the ORD, many 
professionals will argue that they should be a mandatory part of the ORD. 
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4. Contracting 

As the previous section indicates, to attain a desired combat capability, or 
operational thresholds and goals, requirements must be communicated in the ORD in 
clear operational terms, a responsibility of the Combat Developer. The reliability 
objectives must then be translated into quantifiable and verifiable contractual terms 
traceable back to the operational requirements. The Materiel Developer must adequately 
translate the system operational terms into viable contractual terms understood by all 
parties involved to include the user, the program office, and the contractor so that 
compliance can be adequately monitored and enforced. Previously in the traditional 
acquisition process, the Materiel Developer could insert reliability requirements in the 
system specification and development specifications and then incorporate tasks in the 
statement of work (SOW), allowing the contractor to conduct a disciplined reliability 
program to achieve the requirements (SD-2 “Buying Commercial and .. .”, Ch. 6). 
However, recent policy changes resulting from the military specifications and standards 
reform in 1994 has led to the incorporation of a performance-based approach to reliability 
in Request for Proposals, eliminating the use of “how to” reliability standardization 
documents. 

a. Performance Specifications 
The MNS, AoA, and ORD are provided to the Materiel Developer 

(MARCORSYSCOM) for performance specification development, or the translation of 
user requirements into performance specifications that should be understandable to 
potential contractors. Performance specifications eventually become major pieces to the 
Request for Proposal (RFP) and the contract, and thus, they are to clearly state what the 
system must do, how well it must perform, under what circumstances and conditions, and 
identify other constraints. However, performance specifications do not dictate to 


contractors how to achieve the required performance. 


It is important to note that developmental testing is conducted to 
contractual and performance specifications, while operational testing is conducted to 
ORD operational thresholds. “The operational user, the program offices, and the 
contractor often get very confused over the process of translating ORD (operational 


threshold) numbers to contract (performance) specifications and vice versa” (DSMC, 
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“Acquisition Logistics Guide”, p. 10-6). The user or warfighter often has various 
measures highlighted in the ORD that must be translated by the program office into 
performance specifications. Table 3.1 provides a sample of user measurements compared 


to the common contractual reliability specification of MTBF. 


USER OBJECTIVE AREA RELIABILITY (MTBF) 


Increase Readiness Mean Time Between Downing 
Events (MTBDE) 


Increase Mission Success Mean Time Between Critical 
Failures (MTBCF) 


Decrease Maintenance Personnel Costs Mean Time Between 
Maintenance Actions (MTBM) 


Decrease Logistic Support Costs Mean Time Between Removals 
(MTBR) 





Table 3.1. Measures of Systems Readiness. (From: DSMC, “Acquisition Logistics 
Guide”, p. 10-6) 


There must be a clear connection between the defined operational 
reliability requirements in the ORD, created by the Combat Developer and _ the 
performance specifications completed by the Materiel Developer in the terms of the 
contract. Conversion of commonly used operational terms such as MTBM and MTBCF 
must be made to enable translation to parameters that can be specified in contracts as well 
as verified in testing. In doing so, one of the major difficulties is attempting to merge 


contractual reliability and operational reliability. 
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CONTRACTUAL RELIABILITY | OPERATIONAL RELIABILITY 


* Used to define, measure and evaluate * Used to describe reliability performance 
contractor’s program when operated in the planned environment 


* Derived from operational needs * Not used for contract reliability 


esd requirements (requires translation 
* Selected such that achieving them allows qu Creatures ) 


projected satisfaction of operational * Used to describe the required level of 
reliability reliability performance 


¢ Expressed in inherent values * Includes the combined effects of item 
design, quality, installation/repair 
environment, maintenance policy, repair, etc. 


TYPICAL TERMS: 


¢ Accounts only for failure events subject to 
contractor control 


* Includes only the design and 
manufacturing characteristics * MTBM (Mean Time Between 


TYPICAL TERMS: Maintenance) 


¢ MTBF (Mean Time Between Failure) © MER D (Mean. Tine Debyeet Demand) 


7 . ¢ MTBR (Mean Time Between Removal) 
¢ Mission MTBF (sometimes called 
MTBCF) ¢ MTBCF (Mean Time Between Critical 
Failure) 





Table 3.2. Contractual vs. Operational Reliability. (From: Reliability Engineers 
Toolkit: Rome Laboratory) 

b. Source Selection Factors 

The Marine Corps also has the opportunity to use reliability as a factor in 
source selection, arguably the most important contractor motivational factor. In source 
selection for a modified or new system, reliability must be singled out as a specific 
evaluation sub factor. Reliability should be a performance requirement used in the 
solicitation process. In other words, reliability plans and goals should always be a source 


selection evaluation sub factor. 


In the solicitation process, Request For Proposals (RFPs) include a strict 
minimum number of critical performance criteria that force contractors to meet the 
desired program objectives. The desired reliability and cost objectives can be used as a 
management or leveraging tool that forces contractors to meet such objectives. Because 
potential suppliers are competing for a contract, there is a natural tendency for contractors 
to emphasize their strengths while concealing their weaknesses. While it is often useful 
to utilize contractor testing results, it is important to ascertain their capabilities through 
probing, questioning, and eventually, independent military testing as will be discussed in 


an upcoming section. 
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Cc. Contracts, Clauses, Warranties, and Incentives 

After reliability requirements have been established, “the apportioned 
values (MTBF, MTTR, and/or relevant criteria) should be included in appropriate 
sections of procurement specifications, critical item specifications, and contractor end - 
item specifications” (DSMC, “Designing Quality Into Defense Systems”, p. 17). The 
contractor and designer must clearly understand every critical requirement the system 


must meet so that if needed, trade-offs can be executed based on government priorities. 


While predicted reliability typically comes from contractor claims, the 
DoD needs some confidence level that it is a good system of merit for predicted 
reliability. The Materiel Developer must attempt to contract to a given or specified 
reliability confidence level or to a commitment to a specified target operational 
availability in an effort to hold contractors accountable to their original reliability 
estimates. When dealing with contractors’ predicted reliability, the null hypothesis that 


the estimate is incorrect should be ass umed until proven otherwise. 


Additionally, the contracts resulting form the source selection should have 
incentive clauses related to the level of reliability achieved and verified. Warranties can 
be utilized to hold contractors responsible for sustaining in the operational environment, 
the performance levels which have been contractually agreed to. Then, if the contractor 
does not meet the contractual reliability goals, reliability shortfalls should be considered a 
latent defect. Additionally, incentives such as cash rewards can be used to motivate 
contractors to exceed minimum program requirements and predetermined thresholds for 
reliability. However, the use of contract warranties and incentives sometimes imposes 
unrealistic data collection demands on the operational user and field maintenance 
organization, making it difficult to enforce the warranty provisions. The operational 
scenario must be evaluated to determine if warranty conditions are practical. 
Unfortunately, in the past, 

PMs often disregard(ed) logistics contract considerations, such as 
identifying logistics deliverables and creating the logistics input to the 
Statement of Work (SOW), as long-term issues that are less important than 
the immediate problems. As a result, logistics concer ns are (were) often 


deferred for later resolution (DSMC, “Acquisition Logistics Guide”, p. 17 - 
8). 
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One of the more recent trends has been experimentation with Contractor 
Logistics Support (CLS), which has shown indicators of lower costs and/or increased 
readiness. Under CLS, the performance of maintenance and/or materiel management 
functions for DoD systems is conducted by commercial activities. A discussion of the 


benefits and challenges of CLS are beyond the scope of this thesis. 


Another recent initiative has been the use of Performance Based Logistics 
(PBL) and Performance Based Payments (PBP). This strategy is a method of providing 
financing to contractors, performing under fixed-priced contracts, where performance 
based payments are given upon the achievement of specific events or accomplishments 
that are defined and valued in advance by the parties to the contract, rather than being 
tied to and based upon incurred costs of performance (DoD Users Guide to Performance 
Based Payments, Chap 1). It is an integrated acquisition and logistics process for buying 
weapon system capability and instead of buying set levels of spares, repairs, tools, and 
data, there is a focus on buying a predetermined level of availability to meet the 
warfighters’ objectives. In PBL, the contract requirement is specified in service terms. 
For example, the number of hours at a given cost per hour and customer response-type 
metrics such as availability may be used to describe the service. When properly 
incetivized, the PBL provider strives for continuous improvement in reliability to 


eliminate his maintenance efforts altogether. 
The bottom line remains that, 


the well-meaning but ineffectual philosophy often applied to reliability — 

“we will do the best we can’ should be replaced by a contractual obligation 

in the form of quantitative system reliability requirements that forces 

contractors to consider reliability equally with other system parameters 

such as performance, weight, cost, etc (Kececioglu, ‘Reliability 

Engineering Handbook,” Chap. 15). 

To do so, contracts and contract warranty clauses must be specific while 

the user, the program office, and the contractor must understand and agree to the 
reliability terms in both the ORD and contract specification. Ultimately, reliability and 


logistics program success are a direct reflection of contract success. 
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5. Conceptualization, Design, and Development: Systems Engineering 
Process 


System effectiveness and cost are the drivers in design decision, and given the 
trend towards the development of increasingly complex weapon systems, it is obvious 
that reliability cannot be ignored and left as a matter of chance when considering design. 
Instead, reliability must be consciously and proactively built into systems through 
effective design and manufacturing practices. The method for doing so is the systems 
engineering process (SEP), which is used to translate operational needs and requirements 
into a system solution that includes the design, manufacturing, test and evaluation, 
support processes, and products. This includes transforming operational needs and 


requirements into an integrated system design solution through concurrent considerations 
of all like-cycle needs. 


A major goal and function of the systems engineering process is the achievement 
of a proper balance cost, schedule, risk, and performance (to include readiness and 
supportability). To do so, supportability analyses are conducted as an integral part of the 
systems engineering process, beginning at program initiation and continuing throughout 
system development. Supportability analyses form the basis for related design 
requirements included in the system specification and for subsequent decisions 
concerning how to support the system in the most cost-effective manner over its entire 


life cycle (DSMC, “Acquisition Logistics Guide”, pp. 3-10 — 3-12). 


The system engineering process is an iterative interdisciplinary problem solving 
methodology that allows the Government and the contractor to create an integrated and 
life cycle balanced set of system product and process solutions based on Government 
performance specifications and system requirements. The process serves to determine 
critical interfaces for system integration by progressively decomposing system 
requirements nto performance specifications and defining all subsystems, assemblies, 
and parts. As a result, the SEP assists in verifying that the system design meets user 
requirements. While the system engineering process is typically applied at the prime 
contractor level, relevant requirements are passed down to _ the 
subcontractor/supplier/vendor levels. Figure 3.3 illustrates the iterative nature of this 


process. 
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Figure 3.3. Systems Engineering Process. (From: DSMC Program Managers Tool Kit, 
p. 65) 


Application of the system engineering process to reliability design is 
accomplished through a structured process of functional analysis, design synthesis, 
alternative exploration, trade-off evaluation, and decision making which is iterated 
throughout the design process to achieve the desired levels of performance. Maximum 
benefit accrues through the integration of reliability into the system engineering process 
during early development activities since most of the system life cycle costs are 
determined in the early phases of development. The SEP is based on the Integrated 
Product Process Development (IPPD) framework, which is a management technique that 
simultaneously integrates all essential acquisition activities through the use of multi- 
disciplinary teams to optimize the design, manufacturing, and supportability processes. 
The multi-disciplinary aspect of SEP serves as an effective way to get the various 
disciplines working together. Thus, systems engineering programs are required by Do D 


5000.2-R for all Acquisition Category (ACAT) programs. 


The balanced integration of logistics considerations into the systems engineering 


process is imperative from the onset. System reliability, maintainability, and 


ai! 


supportability must be key elements of the tradeoff and design criteria in each stage of 
the process as design considerations will inevitably be in conflict with reliability, 
maintainability, and supportability goals. When such conflicts do occur, the latter goals 
must be considered equally with acquisition cost, schedule, and performance. The 
logistician must be a principal player in the development process as indicated in the 


below excerpt from MIL HDBK-502. 


Unfortunately, acquisition logistics (supportability) objectives often 
conflict with other design objectives like speed, range, size, etc. How is 
this inevitable conflict resolved? Early in the process, the issue of 
tradeoffs must be raised during the analysis of proposed concepts. Careful 
use of tradeoff studies will guide the engineers and the logisticians in 
finding the optimal design -- one which balances design objectives with 
supportability requirements. Tradeoffs are an essential part of the design 
process. 


The result of this early collaboration between engineering and logis tics 
personnel is a specification that prescribes performance requirements to be 
achieved. 


The challenge is to ensure that supportability is integrated into the 
program from the beginning phases. The early design phases of a project, 
when things change rapidly, may seem of little interest to logisticians, and 
their attendance at engineering design reviews may seem a waste of time. 
Actually this period has far reaching logistics impact. During this phase 
the logisticians can use the leverage of early program involvement to 
identify approaches that will significantly lower life cycle costs. They may 
be able to catch an exorbitantly expensive material or time-consuming 
maintenance process before it has become integrated into the system (MIL 
HDBK-502, 6.2.1). 


6. Test, Production, and Verification 


One must learn by doing a thing; for though you think you know it, you 
have not certainty until you try. - Sophocles 


Once a system has been selected, it is imperative to demonstrate, through testing, 
that system capabilities meet contract specifications and satisfy mission needs. 
Specifically, as the proceeding sections will indicate, reliability demonstrations and 
consequential logistics and supportability factors must be included as part of the testing, 
production, and verification of new weapon systems. Unfortunately, demonstration of 


required reliability performance levels prior to system fielding is often a challenge. 
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Because the logistical support system will be built upon the accepted reliability 
estimates, the verification of reliability figures is crucial. It is during testing that DoD 
organizations must validate the contactors’ reliability estimates in an effort to avoid 
future unexpected life cycle cost, supportability, and readiness problems as weapon 
systems mature. Based on system design and its reliability and maintainability 
predictions, the PM office will determine the number of spares of each particular type 
that will be purchased, what support equipment will be used, whether new equipment will 
be procured, the types of skills needed and the varying skill levels required as well as 
other manpower considerations, funding requirements, and POM considerations. If the 
USMC is basing its Integrated Logistical Support Packages (ILSP) upon initial contrac tor 
reliability estimates prior to fielding, it is imperative to have accurate reliability 
estimates. Unfortunately, contractor reliability estimates (of systems and_ their 
components) are sometimes far different from the actual achieved reliability of fie lded 
systems, causing possible catastrophic effects, readiness degradation, or enormous and 
unexpected Life Cycle Costs which eventually create additional need for O&S dollars in 


later years. 


Testing (to include reliability testing) serves several general purposes: 1.) to 
gauge the progress being made when a concept is being translated into an actual product; 
2.) to mature the system by revealing design and process deficiencies so that corrective 
action may occur when it is least costly to fix; and 3.) to determine compliance with the 
requirement and determine operation suitability through formal qualification or 
demonstration testing prior to fielding. There are many types and levels of technical and 
operational tests that are available and used by both contractors and the government. 
While discussion of such tests are beyond the scope of this thesis, some of the common 
tests include but are not limited to: simulations, environmental stress testing, accelerated 
life testing, reliability development/growth testing (RD/GT), reliability qualification 
(RQT)/demonstration testing (RDT), developmental test and evaluation (DT&E), 
operational test and evaluation (OT&E), early user test (EUT)/Limited User Test (LUT), 
initial operational test (IOT), life fire test and evaluation (LFT&E), follow -on test (FOT), 
and many more. For general background purposes, the next sections will briefly examine 
DT&E and OT&E, the two most general categories that of DoD testing. Table 3.3 serves 
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to further distinguish between developmental and operational testing while 


complementing the proceeding sections. 


DEVELOPMENTAL TEST & OPERATIONAL TEST & 
EVALUATION EVALUATION 


+ Technical performance « Designed to obtain operational 
measurement effectiveness / suitability data 


¢ Developing agency responsibility | « Operational Test Agency 
(PM) Responsibility (MCOTEA for 


¢ Technical personnel tee 


¢ Limited test articles / each test * “Typical” user personnel 


¢ Realistic combat environment and 
threats 


¢ Controlled environment 


¢ All types of test articles / 


prototypes ¢ “Production Representative” test 


articles / LRIP items 
¢ Government / contractor 


. e i j 
involvement Contractor involvement restricted 





Table 3.3. DT&E and OT&E Comparisons. (From: DSMC PM Toolkit, p. 51) 


a. Developmental Test and Evaluation 

The overall goal of developmental testing is to determine whether the 
weapon system meets the technical contract and performance specifications. DT&E is a 
method for the PM to make the system work, to verify contractor claims and predictions, 
and to influence the system design. Such testing assists in the development and 
maturation of products, product elements, and support processes and is utilized to verify 
the status of technical progress, verify that design risks are minimized, and certify 
readiness for initial operation testing. While both contractors and Government personnel 
are involved in DT&E, the tests are generally accomplished by engineers, technicians, or 
operator-maintainer test personnel in a controlled environment to facilitate failure 


analysis. 


The feedback provided by developmental testing allows those personnel 
involved in the systems engineering process to analyze the test results and implement 


required adjustments before testing again. As expected, reliability engineers and 
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logisticians play a critical role during DT&E through the IPT process. However, the 
Program Manager ultimately controls the DT environment and is provided with the data 
throughout the testing cycles, enabling the PM to make informed managerial decisions 
that affect the reliability of the final product. Developmental testing identifies 
capabilities and limitations of alternatives and comparisons of candidates. The PM 
typically is forced to make cost-performance trade-off decisions before eventually 
certifying that the system is ready for operational test and evaluation (OT&E). 

b. Operational Test and Evaluation 

Operational testing is the field test for any system or key component of the 
weapon system, conducted under realistic conditions, to determine the operational 
effectiveness and suitability of the system for use in realistic combat conditions by 
typical military users. Operational testing should determine whether minimum 
acceptable operational performance requirements (ORD thresholds) have been satisfied. 
Unlike developmental testing, operational testing is conducted by independent military 
test organizations not beholden to the program office, which represent the customers or 
combat units that will ultimately use the systems. As a result, operational testers 
typically have more independence than developmental testers as they provide their results 
to Congress as well as to senior officials in the services and the Office of the Secretary of 
Defense (GAO, “A More Constructive Test . . .”, p. 11) 

Cc. Testing Summary 

Despite what category or level of testing is being conducted, credible and 
properly designed tests must be addressed, conducted, and properly evaluated early in the 
development process for results to be useful. However, weapon system programs have 
traditionally suffered from persistent problems associated with late or incomplete testing. 
While discovery of problems in any complex product (through testing) is a normal and 
desired part of the developmental process, surprises in testing or repeated occurrences 
often polarize organizations into proponents and critics of programs. It is difficult for 
weapon system programs to compete for approval unless the system offers significantly 
better performance over other systems while remaining within available funding and 
scheduling constraints. As a result, there are greater incentives for PMs to “accept 


immature technologies and make optimistic assessments about what can be accomplished 
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with limited resources.” Test results tend to become scorecards that demonstrate whether 
the program is ready to proceed or to receive the next increment of funding. In the DoD, 
unlike in the commercial sector, testing and evaluation is more for the benefit of the 
testers and decisionmakers above the program manager. Thus, managers often have 
incentives to postpone difficult tests and to limit open communication about the test 
results (GAO, “A More Constructive Test .. .”, p. 8-9). 

7. Maintaining Reliability of Fielded Systems 

Managing reliability does not end with OT&E and fielding of the system, and 
instead, reliability must be continually monitored and assessed for potential 
improvements and efficiencies in support of meeting Marine Corps life cycle cost and 
readiness objectives. In fact, once a system is fielded, reliability assessment should 
become a permanent part of sustainment activities conducted by Program Management 
Offices as well as other Life Cycle Management organizations. To be successful, 
reliability growth must continue during the customer-use phase by coordinating feedback 
from the warfighters to the suppliers and by supporting necessary corrective actions. Part 
of Phase III (Production, Fielding/Deployment, & Operational Support) responsibilities 
include ensuring fielded systems continue to meet mission requirements throughout their 
planned life cycles. Specifically, critical systems and components should be identified 
where low reliability rates are degrading readiness and causing unnecessary support 


costs. 


The basic policy of DOD is to hold contractors responsible for quality of the 
products through quality assurance programs. Quality assurance is defined in DODD 
4155.1 as “a planned and systematic pattern of actions necessary to provide adequate 
confidence that material, data, supplies, and services conform to established technical 
requirements and achieve satisfactory performance” (DSMC, “Designing Quality Into 
Defense Systems”, p. 8). This obviously requires a plan and action, which must be based 
on the quality requirements as outlined in the ORD. To do so, it is recomm ended that a 
program use the reliability requirements stated in operational requirements, or those 
resulting from trade-off analysis, as a baseline for reliability assessment to be compared 
with actual achieved field reliability. However, the difficulty remains in collecting, 
interpreting, comparing operational (achieved) reliability with contractual reliability 
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measurements as illustrated in the previous Table 3.2. Aside from the essential collection 
of achieved field data, original contractor estimates and requirements must be retained for 
comparison. It may not be surprising to find that such documentation is not typically 


retained and is difficult to locate. 


An example of the difference between inherent (or potential) reliability and 
achieved value is shown graphically in Figure 3.4. The operation and maintenance of 
equipment in the field can induce these effects by stressing systems beyond predicted 
levels. Additionally, the true achieved reliability can be obscured by scheduled and 
unscheduled maintenance actions and the corresponding incorrect administrative actions. 
Operational contributors to such overstresses include neglect, unfamiliarity, carelessness, 
and mission constraints. Maintenance actions can also induce defects in otherwise 
satisfactory assemblies; foreign objects introduced, fasteners improperly engaged, 
contaminants introduced, improper part replacement, improper lubricants, etc. While a 
major effort is made in operations to reduce the effects of reliability degradation caused 
by maintenance, the designer should consider the risks of field maintenance and 
minimize the characteristics of the design that are susceptible to operationally induced 
reliability deterioration. Equally important, reliability predictions should be made on 
realistic operational projections for degradation. (DSMC, “Designing Quality Into 
Defense Systems”, p. 28) 


However, it can be argued that reliability requirements can and should be 
established for each phase or product life cycle of a system such as storage, 
transportation, installation, standby, and operation. Therefore, a realistic reliability 
requirement must account for all application environments and the time proportions 
expected in each, and an apportionment of the requirement across the life cycle phases 
accounts for deterioration in each phase (Reliability, Maintainability, and Supportability 
Guidebook, SAE, p. 75). Ultimately, perhaps contractors should attempt to account for 
all elements contributing to the combined failure rate (Table 2.1) and provide the 
government with a confidence interval for a predetermined readiness performance in the 
form of operational availability. Such ideas are open to dispute and will be discussed in 


the upcoming analysis chapter of this thesis. 
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Figure 3.4. Sustaining Reliability in Production and Service. (From: DSMC, 
“Designing Quality into Defense Systems”, p. 28) 


D. CHAPTER SUMMARY 


Beginning with the initial requirements generation, through each iteration of the 
systems engineering process, and ultimately during post-production, reliability must be 
planned for, monitored, accessed, and improved during the maturation of a weapon 
system. The greatest impact on life cycle cost and future operational availability are 
realized during the early phases of system design and development, and thus, logistics 
and the design for supportability must be inherent within early system design 
development if the results are to be cost-effective. The Department of Defense (DoD) 
must continue to strive for the integration of acquisition and logistics in an effort to 
ensure a superior product support process by focusing on total ownership cost, 
supportability as a key design and performance factor, and logistics emphasis in the 
systems engineering process (DSMC Acquisition Chart, 2001). Reliability must be the 
focus of such core planning. Fortunately, as discussed in this chapter, many opportunities 


exist throughout all phases of the acquisition process to effectively address reliability. 


The next chapter examines reliability management techniques and methodologies 


utilized by program management offices as well as common issues and inhibitors 
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associated with reliability management. The data was collected via an electronic survey 


and the results are presented in aggregate form, organized by general themes. 
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IV. MANAGING RELIABILITY IN ACQUISITION PROGRAMS 


A. INTRODUCTION 

This chapter explains the methodology utilized for data collection and presents 
the data gathered to address the primary and subsidiary research questions. The data 
collected relates to a variety of technical, programmatic, managerial, and procedural 
issues and concerns; common practices; and acquisition experiences that relate to 
reliability. The data presented reflects the actions, experiences, and perceptions of the 
acquisition workforce that deals with reliability management issues within the Marine 
Corps. The primary source of data collection was a web-based reliability performance 
survey that was distributed to targeted program management offices via the Acquisition 
Logistics Office at Marine Corps Systems Command. The survey was a modification of 
a similar survey, previously distributed to a Program Executive Office within the Army 
acquisition community, as well as from the literature review and the background research 
on reliability, described in Chapters II and III. A copy of the survey can be found in 
Appendix B. It should be noted that the survey data from the responding program offices 
is presented in aggregate form, organized by general themes, and summarized in tables 
created by the author. 
B. METHODOLOGY 

In an effort to determine the current environment for reliability management 
within the Marine Corps acquisition community, the researcher administered an 
electronic survey to various personnel within the Program Offices of specific 
critical/pacing end items. The survey directions requested attention be given by upper 
level management personnel such as the PM or deputy PM. Respondents included 
Program Managers, Program/Project Team Leaders, reliability engineers, and heads of 
the logistics engineering divisions. The questions posed were intended to emphasize the 
perspective of program management leadership on the varied tasks involved with 
reliability management. In addition to the qualitative-natured questions concerning 
management and procedural issues, numerous quantitative questions were included to 
determine and compare required reliability, estimated or predicted reliability, and 


achieved reliability. As a supplemental source to gain insight into reliability management 
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issues, interviews were also conducted with current acquisition professionals familiar 
with program and reliability management, to include personnel from various program 
offices, the test community, reliability management disciplines, various studies and 
analyses branches, and personnel from academic disciplines. 

1. Program Demographics 

The systems originally intended for research were limited to mature 
critical/pacing end items included in the Quarterly Readiness Reports to Congress (M1A1 
tank, AAV family of vehicles, LAV family of vehicles, 5-ton truck family of vehicles, 
HMMWYV family of vehicles, MK-48 LVS Power Unit, and M198 Howitzer). All the 
programs are Acquisition Category (ACAT) level I and are part of the Marine Corps 
ground equipment inventory. However, it should be noted that some of the systems were 


procured with the Army acting as the executive agent. 


Legacy systems, as opposed to systems in development or recently fielded 
systems, were targeted due to the expected availability of achieved field reliability data, 
which was to be compared with required and estimated reliability. Due to the operational 
age of the systems, replacement systems are currently in development for several of the 


systems. 


As a result of non-participation by a significant portion of the targeted programs, 
additional willing participants, largely from the AAAV program office, offered input to 
the qualitative portion of the survey. However, due to the early stage of the AAAV 
development, the quantitative data questions on estimated and achieved field reliability 
were not applicable to the program. 

2. General Survey Question Themes 

The research was intended to evaluate how weapon system reliability 
performance is managed throughout the acquisition process by identifying common 
inhibitors and enablers of effective reliability management, why they occur, lessons 
learned, and potential methods for mitigating the inherent risks. To do so, the survey 
consisted of 37 primary questions, some with subparts, which focused on five major 
themes, developed for the purpose of this thesis, and listed below: 

e management approach to reliability 


e determining and documenting reliability requirements 
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e contracting and incentivizing for reliability 

° reliability testing 

e comparing and assessing required, estimated, and achieved field reliability 

Collectively, these themes correspond to issues addressed in the thesis research 
questions. 


3. Data Presentation 


In an effort to obtain disclosure of all issues associated with reliability 
management, respondents were permitted to provide information under the premise of 
non-attribution. Likewise, survey instructions specific ally stated that all program 
responses would be presented in aggregate form. Responses were received from only 
three of the seven programs originally solicited for participation. As a result, the 
researcher sought additional programs for participation to gain further perspectives on 
reliability management issues. The additional programs were incorporated through their 


survey responses, interviews, and email correspondence. 


The subsequent sections provide the data for this research and serve as the basis 
for analysis in Chapter V. The survey responses and corresponding data are organized 


into the previously mentioned five major themes. 


While all the themes have subparts, each theme is generally presented in the same 
fashion. First, the purpose of the survey questions within that main theme is addressed. 
Next, narrative summaries of responses, data tables, illustrative examples of reliability 
management experiences, responses in the form of quotes, or a combination of such, are 
presented. Lastly, the author summarizes the responses and data to exemplify challenges 
that program managers face when dealing with reliability issues of their systems. 

C. MANAGEMENT APPROACH TO RELIABILITY 

Purpose of Theme: The first series of survey questions focus on how reliability 
and its associated risks are managed. The questions asked the program offices: 1) what 
they perceived to be the key factors that contribute to reliability problems in a program; 
2) how reliability performance is managed within a PMO in terms of roles and 
responsibilities, documentation, and activities utilized to recognize and evaluate potential 
system failures; and 3) their opinion and understanding of the amount and adequacy of 


DoD and USMC policy and guidance on reliability. 
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1. 


Key Factors Contributing to Reliability Performance Problems 


Given a list of fifteen common prevailing issues, the survey participants were 


asked to rank order what they perceived to be the top five factors that contributed to 


reliability problems in program management. Respondents were also given the 


opportunity to nominate “other issues” and rank them relative to the fifteen issues 


provided in the survey. Table 4.1 provides a compilation of the top responses, presented 


in an overall composite order of merit ranking, from the most significant factor to the 


least significant. 


Survey Responses: 


1. 


TOP RATED FACTORS CONTRIBUTING TO 
RELIABILITY MANAGEMENT PROBLEMS 


Traditional test & evaluation RAM metrics are not supported by 


maintenance data sources (unable to make a valid comparison b/n RAM 
requirements and estimates with achieved field data) 


2. 


Too much pressure to field systems rapidly (schedule goals outwe igh 


reliability) 


3. 
4. 


Need more qualified reliability personnel in PMOs 


Unrealistic reliability requirements with inadequate rationale 


- Poor reliability planning and growth planning (test too late ) 


. Missing or poorly written ORD reliability requirements 


5 
6 
7. 
8 
9 


Insufficient reliability testing to verify requirements 


. Contractor not designing for reliability sufficiently above the requirement 


. Too much pressure to field system cheaper (cost goals outwei gh reliability) 


10. Not consistently improving reliability after fielding 





11. Inadequate or vague policies and guidance (need updating) 


Table 4.1. Top Reliability Management Problems as Perceived by Survey 


Respondents within the Acquisition Community. 


Given the opportunity to nominate their own factors affecting reliability 


management, several respondents did so, providing the following comments: 


PMs are not provided the resources or authority to impact reliability 


Engineers pay more attention to meeting performance requirements than 
to reliability requirements when they should be considered more equally 


Traditionally, PMs have been evaluated on cost, schedule, and 
performance. Thus, reliability often got pushed aside 
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° The PMs, the Primes, and all the members of the IPTs should be evaluated 
on readiness performance that have force of law 


e Currently, PMs are graded on cost (testing costs and costs to field), 
schedule, and performance in accordance with the Defense Acquisition 
Executive Summary (DAES) vs. SUPPORTABILITY, in the form of LCC 
or some target A, 


e Dollars drive the train (acquisition process) instead of requirements 
Summary: The top three inhibitors to effective reliability management, as ranked 
by the survey respondents, were clearly identified as problematic as all of the r espondents 
chose all three of these choices as one of their top five ranked issues. Interestingly 
though, twelve of the fifteen (survey -provided) choices received two or more votes, and 
five of the fifteen choices received at least one vote as the top inhibitor to effective 
reliability management. 
2. Managing Reliability in Acquisition Programs 
The next series of questions deal with program management approaches to 
reliability, to include the perceived roles and responsibilities of dealing with reliability, 
formal documentation of a reliability program plan, and activities utilized to recognize 
and evaluate potential system failures. 
a. Reliability Roles and Responsibilities 
Survey participants were asked, “who within the organization was 
primarily responsible for program reliability activities.” The author desired to determine 
how PMs delegated responsibility for reliability activities and whether there was a 
consistent managerial approach in doing so. If the respondents indicated that reliability 
activities were conducted within the context of an Integrated Product Team (IPT), 


responders were asked if the IPT was formally chartered. 


Survey Responses: Responses varied throughout the programs without 
any overwhelmingly unified response. The most common responses indicated that either 
Logistics/Supportability Team Leaders or Project Team Leaders had been delegated 
primary responsibility for reliability issues, each receiving two responses. Only the 
AAAV program indicated the use of reliability IPTs. Additionally, two programs 
recognized that the PM had ultimate responsibility while delegating reliability issues to 


Team Leaders and others. Lastly, one program respondent could not identify an 
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individual or team that had overarching responsibilities associated with reliability 
activities, choosing the “no one specifically” survey option, possibly suggesting a shared 


responsibility amongst multiple sources. 


A former PM commented that PMs manage by exception and without a 
specific problem or issue, reliability and the other engineering disciplines are managed 


through empowerment of technical experts (Masiello, p. 39). 


Summary: Responses varied as to the individual or group primarily 
responsible for program reliability activities, and responses included the PM, Reliability 
IPTs, Logistics Team Leaders, Project Team Leaders, and in one case, no one 
specifically. No survey responses indicated that primary responsibility for reliability fell 
upon the prime contractor, test team leader or testing activities, system engineering team 
leader, or the Logistics Management Specialists (LMS). Overall, PMs seemed to rely 
upon reliability competency outside the program through matrix support. 

b. Documenting a Program’s Reliability Approach 

Survey participants were asked, “how the system reliability program and 
the corresponding management approach were formally documented within the 
program(s).” Choices included: reliability program plan, contract statement of work 
(SOW), test and evaluation master plan (TEMP), single acquisition management plan 


(SAMP), no formal reliability management plan, or other. 


Survey Responses: Of the responses received, half of the programs 
indicated that there was no formal reliability program plan. One respondent noted, “there 
is no requirement for PMs to have a formal program or an overarching document 
describing the activities.” Of the programs which had a reliability plan, most relied on 
the contract SOW or TEMP to address: 1) how they intended to ensure reliability was 
treated @ a high priority objective, 2) methodologies and plans for measuring and 


achieving reliability, and 3) the resources needed to execute the plan. 


C. Activities/Tools Used to Evaluate Potential Failures 
There are numerous test and design tools available to program offices and 


contractors that help to ensure that reliability is “designed-in,” early in the program. 
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“Designing-in” reliability upfront reduces risk and is less costly than finding design 


discrepancies during later stages of testing, evaluation, and operational use. 


By inquiring as to which “activities that the program(s) implemented to 
recognize and evaluate potential failures and causes,” the author’s intent was to 
determine the risk mitigation techniques which programs and contractors employed to 
address reliability achievement. The survey asked participants to identify all the testing, 
engineering, and other technical methods used in their respective programs. A list of 
fifteen common testing, engineering, and other technical methods and techniques used to 
determine and evaluate potential failures and their causes was provided for survey 
respondents to choose from. Additionally, participants were provided the opportunity to 


list any other methods utilized by their programs. 


Survey Responses: As expected, developmental and operational testing 
played a major role in the development of all programs. However, the extent to which 
programs failed to utilize other reliability risk mitigation techniques to determine 
reliability achievement was rather astounding. There was one outlier, which was the only 
system examined that is currently in development. While the AAAV program 
respondents indicated the use of all fifteen techniques listed, the other program 
respondents either had very scarce use of the tools or were not aware whether the original 


program staffs and contractors had used the techniques. 


Each program indicated that it utilized only one of the following 
techniques: Environmental Stress Screening (ESS), reliability modeling, FMECA, 
Reliability Development/Growth Test (RD/GT), or FRACAS. Additionally, no program 
indicated the use of reliability allocation, fault tree analysis, probabilistic failure 
assessment, reliability qualification test, PFMEA, Weibull analysis, physics of failu re 
(POF), or a parts control program. One reliability engineer indicated, “we list all of the 
tools that we think will be useful, knowing that PMs will cut many of them, citing fiscal 
constraints” (Masiello, p. 47). 


Summary: Many program representatives were either not aware of the 
specific techniques utilized to ensure reliability was “built-in,” or the original staffs did 


not actually use the available tools. However, there was a common consensus to test 
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early and often, and use knowledge of reliability growth to implement corrective action. 
All PMOs reported using some form of failure analysis as an integral part of the design 
process, and there was a consensus that the use of such tools that incorporate reliability 


prediction and achievement into system design was beneficial. 


The reader should be reminded that in most cases, the survey respondents 
were not the original PMO staff, and the respondents may be aware of which techniques 
were utilized only by reviewing any existing documentation that was retained before their 
arrival. It is assumed that much documentation from the original staff or the contractor 
was no longer available. The assumption that the legacy systems did not take advantage 
of the reliability analysis tools may be invalid. In reality, many of the programs may 
have utilized the tools more than indicated in the survey, and the respondents were not 
aware of the previous staffs’ or contractors’ actions. 

3. Existing Policy, Regulations, and Guidance on Reliability 

The author wanted to determine the level of existence as well as the level of 
awareness of reliability policy, regulations, and procedures. Likewise, the author desired 
the opinions of the acquisition community as to whether the existing regulations and 
guidance were sufficient to help PMOs manage reliability performance in their programs. 
The questions posed to survey participants were, “Are you aware of any specific DoD or 
Marine Corps policy/regulations regarding weapon system reliability management? And, 
do you feel that existing policy and regulations on reliability provide adequate 


guidance?” 


Survey Responses: Six of the seven respondents that chose to answer this 
question stated they were unaware or unsure of any policy or regulation regarding 
reliability management. The PM that answered in the positive did not cite a specific 
manual, document, handbook, or policy, and simply stated that it was the “program 
engineer’s responsibility (to be aware of this)” Most responses and interviews 
commented on frustration concerning the lack of useful documented guidance. 
Additional responses are paraphrased or quoted below: 

e I am not aware of any policies that adequately address reliability 


e You’ ve hit the nail on the head with identifying the vague nature of what 
is currently in print 
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e Due to acquisition reform, the Government has steered aware from 
military specs and standards. Also, this makes it difficult to identify 
which regulations and guidance for reliability are applicable at any time. 


Summary: According to the responses, the acquisition community either has little 
guidance or is not aware of guidance concerning reliability management. Additionally, 
much of the guidance is very broad scoped, providing little detail as to specific reliability 


actions to be taken in the acquisition process. 


D. DETERMINING AND DOCUMENTING RELIABILITY 
REQUIREMENTS 


Purpose of Theme: The next group of questions deals with reliability in the 
context of inputs and procedures of the requirements generation process. The purpose of 
the questions was to determine and assess whether a reasonable and cooperative process 
exists between the Combat Developer and the Materiel Developer, if reliability 
requirements were arbitrarily set or not, if the original reliability requirement was 
documented, and if so, where is it documented, what was the reliability requirement, and 
in what terms was it identified. 


1. Influencing Realistic Reliability Requirements 

A common criticism of the acquisition process is that system requirements are not 
adequately defined or are often unrealistic. The challenge is to address the reliability 
requirements in terms of the users’ operational mission needs and success under given 


conditions, with defined mission profiles, environments, and durations (Ryan, p. 47). 


The following questions and corresponding data address the Materiel Developer’s 
ability to influence system reliability requirements, and the level and terms at which the 
requirements were set. Participants were asked, if “the PMO, as a representative of the 
Materiel Developer, was able to influence incorporation of realistic reliability 
requirements into the process.” They were also asked, “what the documented reliability 
or availability requirement was, and in what terms it was measured (i.e, MTBF, MTBM, 


Ao, MTBSA, MTBOMF, MTBEFF, MTBOMA, MTBMAT, etc.)”. 


Survey Responses: In nearly all cases, materiel developer representatives were 


able to provide input for establishing reliability requirements. 
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Ability to Influence Percentage of 
Reliability Requirement Programs Examined 


NO 0% 


Table 4.2. Influence on the Requirements Generation Process. 





The terms in which reliability requirements were identified varied from program 
to program. Respondents indicated the documentation of requirements in the form of: 
Mean Miles Between Failure (MMBF), Mean Miles Between Operational Mission 
Failure (MMBOMBF), Mean Time Between Operational Mission Failure (MTBOMEF), 
Operational Availability (A,), Mean Time Between Unscheduled Maintenance 
(MTBUM), and Mean Time Between Failure (MTBF). 


Additional related responses concerning reliability requirements are paraphrased 


below: 


e Contractors are in business to provide the Government the products and 
services we request. If they fail to do so, they go out of business. The 
question then becomes, are we asking for what we really want in clear and 
concise terms? When a program fails, too many people in this business 
affix blame to the contractors. Instead, I believe that the Government is 
ultimately accountable to the taxpayers. Did we ask for what we needed? 
Did we select the right contractor to do the job? Did we provide adequate 
support and oversight to the project? I realize that very few officials in 
Government are willing to ask such tough questions. 


e MCCDC has the responsibility of creating the requirements, but the PM 
office comments on the requirements and their rational with MCCDC 


e In order to determine user reliability requirements, emphasis must be 
placed on understanding the user’s system readiness and mission 
performance requirements; and translating them into system requirements 
that can be designed, implemented, and verified 


Summary: According to the responses, it appears that programs actively 
participate with the Combat Developer to determine the requirements, including those 


requirements relating specifically to reliability as part of the RAM determination process. 
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Thus, it may be assumed that a reasonable and cooperative process exists between the 


Combat Developer, Materiel Developer, and the user representative. 


A review of the terms in which the reliability requirement is identified varies from 
program to program, indicating that there is not a standard operational terminology in 
which reliability must be expressed. While this likely allows for flexibility, there must be 
an agreement and understanding between the Government and contractor of those terms, 
as further sections will indicate. 

2. Reliability as a KPP in the ORD 

As the survey responses indicated in the previous section, most programs had 
documented reliability requirements, while identifying the specific terms (MTBF, 
MTBM, etc.) used in defining requirement. It then becomes useful to discover where the 


requirements are documented. 


The author hoped to ascertain the relative importance of reliability with respect to 
other performance parameters. Participants were asked if reliability requirements were 
identified as Key Performance Parameters (KPP) in the Operational Requirements 
Document (ORD). Additionally, they were asked whether the requirement was in an 
objective and quantifiable form that contrac tors and the Government could easily agree 


upon. 


Survey Responses: Only two responses indicated the use of a reliability 
requirement as a KPP — one of which had a sister service as the executive agent, and the 
other was the AAAV, which is the only system still under development from which 
survey responses were collected. Meanwhile, none of the remaining legacy programs 
examined included reliability as a KPP. Some responses indicated that their current 
program staff could not locate the ORD due to the time that has passed and the turnover 
of personnel. Additional related responses were: 


e RAM requirements are usually not KPPs. So when there are cost or 
schedule overruns, these are the first to take a hit. 


e Reliability and maintainability, along with performance, should act as 
equal partners in the requirements generation process 


e Test to requirements in the ORD. If reliability is not a KPP in the ORD, it 
gets pushed aside due to other requirements precedence 
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e There seems to be a huge traceability problem. We couldn’t even find the 
(undisclosed program name) ORD until (undisclosed analyst) called an old 
friend at the contractor who had kept a copy. 


Summary: While reliability requirements were typically not identified as KPPs, 
programs agreed that reliability was an important priority that received varying degrees 


of attention. 


As previously mentioned, all of the systems examined were legacy systems with 
the exception of the AAAV. Interestingly, the newest system examined, which is still 
under development, has designated reliability as a KPP. In fact, the AAAV has a very 
specific MTBOMF threshold as a KPP for the Milestone C decision. Additionally, to 
ensure that the requirement is in an objective and quantifiable term that the contractor and 
the Government can agree upon, the AAAV contractor was “given the Failure Definition 
and Scoring Criteria which was the basis of determining whether a failure was an 
operational mission failure.” 

E. CONTRACTING AND INCENTIVIZING FOR RELIABILITY 

Purpose of Theme: The questions and corresponding survey responses in this 
section relate to the role of reliability in the source selection and contracting process. The 
overall intent of this series of questions was to determine how and to what extent 
reliability requirements were developed into contractual agreements. 

1. Reliability as a Source Selection Factor 

Programs were queried as to whether reliability was included as a factor in source 


selection. 


Survey Responses: With the exception of the AAAV, the program respondents 
replied that either reliability was not a factor in source selection or they were not certain 
if reliability was a factor in source selection due to the time that had passed since the 


program was originally contracted and the lack of documentation in the PM offices. 


Summary: While reliability was not a factor in source selection for the legacy 
systems examined, some respondents gave the impression best put by one individual, 
“Reliability, with its impact on O&S costs, should receive critical attention in the market 
investigation, solicitation, and source selection process. Unfortunately, I believe this is 


typically not the case.” 
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2. Reliability Requirements in Contracts 
The second contract related question inquired as to how operational reliability 


requirements in the ORD were translated into contractual requirements. 


Survey Responses: Roughly two-thirds of the participating survey respondents 
indicated the ORD paragraphs relative to reliability were restated in the Statement of 
Work or performance specifications, indicating that the contract requirement was very 
similar to the ORD requirement. One of the oldest systems, which has exceeded its 
intended life cycle by over a decade and a half due to extensive upgrades and Depot 
Level Maintenance, indicated that comprehensive reliability requirements were not 
adequately stated in the original contract. Conversely, the AAAV sets precedence for 
future systems by applying “additional levels of reliability to the contract as the 
performance specifications (in the contract) set the bar a little higher than the ORD.” 


Additional related responses concerning the contractual reliability requirements are 


provided: 

e While predicted reliability typically comes form contractor claims, we 
need some confidence level that it is a good system of merit for predicted 
reliability. We must contract with the Prime (contractor) for a 
commitment to some target A,. 

e We need to make readiness targets contract items 

e It would require contract changes to hold contractors accountable to their 
estimates 

e Reliability objectives should be translated into quantifiable and verifiable 
contractual terms and allocated through the system design hierarchy 

e Contractual requirements should be traceable to operational requirements 


and capable of verification 


e We should adopt the null hypothesis that states the MTBF is not what the 
contractor claims, but rather what the contractor proves 


Summary: In terms of translating user operational requirements to contractual 
requirements, all but one of the legacy systems examined indicated that the contractual 
requirement was very similar to the ORD requirement, and ORD paragraphs relative to 
reliability were simply restated in the SOW or specifications. The remaining legacy 


system stated that the comprehensive reliability requirements were not adequately stated 
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in the original contract. Conversely, the AAAV applied additional levels of reliability to 
the contract. 

3. Contracting Incentives for Reliability 

The use of meaningful contract incentives for achievin g predetermined reliability 
performance is a method to encourage contractors. Survey participants were questioned 
if incentives that are specifically tied to achieving system reliability performance 
requirements were employed in their programs’ contracts, and if so, did the incentives 


achieve their desired effects. 


Survey Responses: The respondents representing the legacy systems indicated 
that contract incentives were not utilized for the original purchases of their systems. 
However, the AAAV program staff cited the use of a Cost Plus Award Fee (CPAF) 
contract, and further indicated that reliability has been used as one of the award fee 
criterion on numerous occasions thus far. Additional comments are provided below: 


e If a contractor does not meet the predetermined reliability goals, it should 
be considered a latent defect 


e We must tie the contractor to LCC through reliability. In other words, we 
must reduce life cycle support costs through reliability warranties and 
incentives 

° Incentives should be created to reward for good systems in terms of 
logistics 


Summary: Of the legacy systems examined, there was no apparent use of contract 
incentives for reliability achievement. 
F. RELIABILITY TES TING 

Purpose of Theme: Test and evaluation activities are a critical part of every 
program as they serve to aid in the development of a system and to verify that the system 
meets specified standards. The questions in the proceeding sections are concerned with: 
1) the adequacy of time and funding allotted for reliability testing during developmental 
testing, 2) general agreement and common understanding on measures to determine 
reliability performance during testing, and 3) the use of IOT&E entrance criteria relative 


to reliability. 
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1. Resources: Time and Funding Constraints 

In an attempt to achieve program objectives, program management requires 
making trade-offs in terms of cost, schedule, performance, and supportability. Programs 
were queried as to whether the amount of time and funding allotted for reliability testing 


during DT&E was sufficient. 


Survey Responses: All but one program indicated an insufficient amount of time 
and funding allotted for reliability testing during developmental testing. However, this is 
not surprising in the acquisition world wher e program offices continuously are forced to 
conduct trade-offs. One program summarized the constrained resource situation by 
stating, “there is never enough time or money because the more time and money 


(available), the more failures that can be uncovered and corrected.” 


Summary: A common perspective relayed by the program offices is a lack of 
time and money to conduct adequate levels of reliability testing which are needed to 
achieve a substantial confidence level of the system reliability. In fact, data from the first 
survey question indicated that too much pressure to field systems quickly and too much 
pressure to field systems cheaply were respectively the second and ninth ranked 
inhibitors to effective reliability management. 

2. Agreement on Reliability Measures for Tests 

The concept of reliability is often used without precise definition, while the 
terminology is non-standard throughout the logistics community and tends to depend on 
the system being developed. However, while creating DoD requirements documentation, 
contract specifications, and test documentation, it is very important that all main concepts 
are addressed in an unambiguous way so that all parties involved (to include the user, 
combat developer, materiel developer, PM, contractor, and tester) understand the terms. 
Survey participants were asked if the user, contractor, tester, and PM all agreed upon the 


method used to determine reliability performance during testing. 


Survey Responses: One of the legacy programs answered affirmatively, stating 
that the agreed upon method could be found in the TEMP. The remaining systems 
indicated that they were uncertain if such an agreement had been made amongst all 


parties. The numerous “not certain” responses are likely a result of the time that had 
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lapsed, the turnover of personnel, and the loss of documentation since the test phases had 


occurred years prior. 


3. Testing to Determine Reliability Requirements Conformance 
a. Initial Operational Test & Evaluation (IOT&E) Entrance 
Criteria 


Operational Test and Evaluation is the final test conducted prior to the 
decision on whether the system will proceed with full rate production. Given the 
significance of this program gate, entrance criteria, relative to reliability, is often 
established to ensure that the system is prepared for Initial Operational Test and 
Evaluation. Meeting reliability entrance criteria commonly involves testing reliability in 
Developmental Testing activities and involves validating required reliability levels. Such 


entrance criteria are most often required by the independent testing organization. 


Survey participants were asked if their respective programs had specific 


IOT&E criteria relative to reliability. 


Survey Responses: A significant majority of the responding programs had 
IOT&E entrance criteria relative to reliability. Furthermore, most criteria were 
established in very specific terms, such as MTBOMF. Notably, the AAAV program 
indicated that the Milestone C criteria, which allows the program to build Low Rate 
Initial Production (LRIP) vehicles for IOT&E, was to “demonstrate system reliability 
within the Growth Curve at 80% confidence through a mix of test data and analysis.” On 
the other hand, one program indicated that IOT&E entrance criteria were not used by the 
sister service executive agent, and one program was uncertain if entrance criteria were 
utilized. 


b. Reliability Demonstration During Developmental and 
Operational Testing 


Programs were asked to what level were their systems’ ORD reliability 
requirements demonstrated during developmental testing, operational testing, and during 
sustainment. Additionally, programs were asked to what level were the contractors’ 
reliability estimates demonstrated during developmental testing, operational testing, and 
sustainment. By posing such questions, the author intended to gain a better 


understanding of the required level of reliability demonstration prior to the program 
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proceeding, whether reliability requirements and contractor estimates were sufficiently 


realistic to be achieved, and whether there was a correlation between success/failure in 


DT&E, OT&E, and sustaiment with respect to reliability performance. Survey test 


related responses are provided below: 


We’ve found they (PMOs) don’t have much documentation from the 
DT&E phase of programs. We’ve had more success finding OT&E 
reports from MCOTEA since these are Government-run and usually 
archived by DTIC or service libraries. 


The DT&E reports were probably submitted to PM staff and ended up 
taken or lost when those people moved on 


It is during T&E that we (Government) must validate the contractor’s 
estimates. Under- or over-estimating reliability will cause limited funds to 
be allocated unwisely 


Once the contractor submits their RAM estimate, it becomes the 
Government’s estimate if we accept it. Therefore, it’s up to us to become 
involved in this process and to conduct independent testing as necessary to 
verify such estimates. 


Demonstration of required reliability performance levels prior to system 
fielding is a challenge 


Estimated or measured reliability should be used to evaluate the design 


Achievement of contractual requirements should be verified through a 
combination of engineering analysis and test results 


Test to (the) requirements in the ORD 


Determination of contractual compliance based on engineering analysis 
without supporting test data can lead to erroneous conclusions 


We must conduct Logistics Test & Evaluation (LT&E) early and 
throughout the DT&E, allowing program office personnel to determine 
what needs to be adjusted to provide the required system support 
throughout the program’s life cycle 


Survey Results: Qualitative responses to this question were limited. None 


of the participating programs were able to identify the level to which the contractors’ 


estimates were demonstrated (during testing phases or sustainment) due to the fact that 


contractor estimates were not retained. 


Additionally, in some cases, the ORD could not be found, meaning the 


original reliability requirement was not retained either. 
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Lastly, most programs indicated that achieved field reliability data 
compiled during the sustainment phases of the respective systems is suspect to error, 
making the comparison of such data with the contractor estimates and original ORD 
requirements (when available) questionable. The proceeding section is devoted to such 
issues and concerns. 


G. COMPARING AND ASSESSING REQUIRED, ESTIMATED, AND 
ACHIEVED RELIABILITY 


Purpose of Theme: Programs that have carefully planned and executed reliability 
management techniques will benefit, in the way of decreased life cycle costs and 
increased operational availability, during sustainment of the system. However, reliability 
performance of a system should be continually assessed throughout its lifecycle. 
Programs often assume reliability to be what the contractor states it to be instead of 
determining progress and compliance with reliability estimates and requirements 


throughout the lifecycle sustainment phase. 


This section summarizes previous questions by compiling and comparing survey 
responses concerning ORD reliability requirements, contractor reliability estimates, and 
achieved reliability. The data is intended to determine whether PMOs and the logistics 
community are adequately engaged in tracking and improving system reliability through 
a systematic process of collecting reliability trend data. 

1. Maintaining Original Contractor Estimates 

Participants were queried as to whether or not the contractor reliability estimate 
was documented, and if so, where was it documented, who retains the documentation, 
what was the estimate, and in what terms was it measured (i.e., MTBF, MTBM, 


MTBOMEF, MMBBF, A,, etc.). 


Survey Responses: All responding programs indicated that either the contractor 
reliability estimates were not documented or the documentation was not retained. One 
respondent stated that the estimates were not retained because the “program was too old”. 
Other related responses are provided: 


e I highly suspect you will have to go to each PM’s office in an attempt to 
locate this information (ORD, contract, reliability rqmt, reliability 
estimate, and achieved reliability) 
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2. 


Where will you get copies of the contractors’ RAM estimates? Does 
anyone retain these documents after the system is fielded? 


Once the contractor submits their RAM estimate, it becomes the 
Government’s estimate if we accept it. 


. we (the Government) must validate the contractor’s (reliability) 
estimate. 


Unfortunately, there is not an annual inspection that checks whether or not 
they are maintaining the data and information. 


Collection and Computation of Achieved Field Reliability 


Calculation of system reliability that is being achieved in the field is necessary in 


order to determine whether the mean time between failures is increasing, decreasing, or 


remaining constant with age. In other words, such data analysis and assessment of 


reliability performance help to determine if equipment is in the “wear -out” phase of its 


life cycle and at the end of its economic useful life. 


Participating program representatives were asked whether the achieved reliability 


of their programs had been computed, and if so, what was the overall achieved reliability 


and in what form was it calculated. It is important to recall from Chapter II that there are 


numerous forms of measurements that relate to reliability achievement, to include MTBF, 


MTBM, MTBOMF, MMBF, A, and others. Table 4.3, in the next section provides the 


results. Additionally, various personnel provided the remarks below: 


USMC maintenance management systems record maintenance activities 
and not system failures, (which are required to calculate MTBF), while the 
systems do not distinguish between preventative and _ corrective 
maintenance activities 


Rather than computing ‘failure rates,’ we are focusing instead on 
‘maintenance rates’ because our AIS systems do not directly record 
failures; MIMMS/ATLASS records maintenance events. We believe that 
maintenance rate analysis is a feasible surrogate for traditional failure rate 
analysis; based on existing data sources, that’s probably the best we can 
do. Even then, we’ ve run into significant obstacles and data quality issues 
with MIMMS. 


Also, any metric based on operational usage data won’t work. The meter 
readings in MIMMS are inconsistent and inaccurate (e.g., odometer 
reading = 999999, mixing engine hours and odometer miles, etc.). 


Utilization data is not consistent 
It is hard to distinguish between corrective and preventive maintenance 
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Summary: Lack of necessary data and related weaknesses in the Marine Corps 
maintenance management data systems prevent the calculation of MTBF. Specifically, 
operational usage data is required to calculate failure rate, a key indicator of reliability 
performance. Thus, the maintenance rate has often been used as a substitute for failure 
rate, but when utilizing MTBM in place of MTBF, it is important to recall that MTBM 
does not distinguish between preventive and corrective maintenance activities. 
Furthermore, issues with data quality derived from maintenance management systems 
also make the calculation of MTBM skeptical. Thus, many professionals believe that the 
next best calculation is the R-rating or readiness rating, known as the SORTS equipment 
condition rating, calculated as follows as per MCBul 3000: 


QOtyPossessed —QtyNotMissionCapable 
pu Li Possessed ~QtyNotMissionCapable 


QOtyPossessed 


The R-rating basically provides a snapshot of operational availability, for which 


the calculation is shown below: 





UPTIME 
FF 
4 uptime = MTBM 7 OT +ST 
° uptime +dowtime MTBM+MDT OT +ST+ALDT +CMT +PMT 
UPTIME DOWNTIME 


The calculation of operational availability is affected by inhibiting factors in the 
logistics system such as administrative delay time, order ship time delays, and delays in 
corrective and preventive maintenance. Additionally, standby time (ST) is not recorded 
and distinguished from operating time (OT). Lastly, the data is extracted from disparate 
sources (MIMMS and ATLASS). 


3. Comparing Required Reliability, Estimated Reliability, and Achieved 
Reliability 


It is important to have a systematic process in place for collecting and comparing 
reliability data for several reasons. The Marine Corps must be able to calculate and 
compare the reliability that is being achieved in the field during post-production with the 
required and estimated reliability in order to determine contactor compliance, 
successfully hold contractors to their estimates, and determine if the user reliability 


requirement is met. 
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The data from the questions in the previous sections were combined in an attempt 
to determine the “reliability gap” of the legacy systems, or the difference between the 
required reliability, contractor reliability estimates, and achieved reliability. 
Additionally, programs were asked what organization(s), if any, have compared and 
assessed actual achieved reliability of fielded systems to the original requirements and 


contractors’ estimates. 


Survey Responses: All of the legacy system respondents stated that no 
organization, internal or external to their programs, has formally or informally 
determined the “reliability gap” for the respective systems up to this point in time. One 
program did note that MCCDC is concurrently conducting a similarly related study 
entitled “Sustainment Consequences of Acquisition Decisions”, which is sponsored by 


MARCORSYSCOM. 


Table 4.3 provides a comparison of three sample programs’ reliability 
requirements, contractor reliability estimates, and achieved reliability, as well as 


participants’ personal perspectives when provided. 


Reliability 43.5 MTBOMF 600 MMBF > 
R i t threshold (not delineated in Not Known 
equiremen ok) 


Contractor peas 
Reliabilit “Estimate not Estimate not “Don’t know if 
eae documented” retained; program! documented: 


Estimate is too old” not retained” 


* “Not certain. The 
(Post-Production) 89 MTBOMF:; Marine Corps does “Ss 80% 


Achieved “data is suspect” | not do a good job 
Reliability capturing this data” 


Readiness ” 








Table 4.3. Reliability Gap. 
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Additional comments concerning the computation and comparison of required, 
estimated, and achieved reliability are provided below: 


e Traditional T&E RAM metrics are not supported by our maintenance data 
sources so how will you make a valid comparison between contractor 
RAM estimates and actual data? 


e Our experience here (Studies and Analy sis branch) is that data is not easy 
to come by 


e Reliability focus should not end with OT&E. Once the system is fielded, 
reliability should become a permanent part of the PM’s and Logistics 
Management Specialist’s (LMS) sustainment activities. Critical system 
and components must be identified where low reliability rates are 
hampering mission accomplishment. 


e Currently using the wrong metrics 

Summary: It is well known that “you can’t manage what you don’t measure,” and 
in general, responses indicate that there is a lack of a systematic process for collecting 
reliability trend data beyond readiness ratings. Furthermore, what data that does exist is 
suspect to error and corruption as a result of the current maintenance management 


automated information systems. 


There was also a consensus that traditional test and evaluation RAM metrics are 
not supported by maintenance management data systems. One may recall that this was 
voted as the top rated inhibitor to effective reliability management, according to the 
responses from the first question of the survey. 

a. MTBM Computation 

A study was completed on 08 June 2002 by Captain Jake Enholm in an 
attempt to formulate a methodology for determining systemic MTBM and equipment 
parts’ (NSNs) failure rates using current warehoused maintenance management data 
drawn from MIMMS/ATLASS II/SASSY data fields. In the calculations used in this 
study, MTBF and MTBM periods were combined as the model used both preventive and 
corrective maintenance actions combined. The results of the study indicated that it is 
sometimes possible to calculate a sample mean or median time between 
maintenance/failure for certain equipment in the Marine Corps. However, the study 


indicated that the accuracy of the analysis is suspect to weaknesses in the Marine Corps’ 
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maintenance management data systems. A full version of the study is available in 
Appendix C. 

b. Depot Level Maintenance Program Effects 

Due to the age of the respective systems, concerns about increasing failure 
rates and increasing costs to maintain, and in some cases, declining readiness trends, 
some of the systems examined have undergone Depot Level Maintenance (DLM) 
programs such as “Service Life Extension Program” (SLEP) or “Inspection and Repair 
Only As Necessary” (IROAN). Programs such as these, which modify, upgrade, or 
change the designs of the systems, as well as overhaul major components, obviously 
affect the reliability of the systems and skew the data that is attempting to be compared. 
Therefore, to take this factor into consideration, survey participants were specifically 
asked whether their programs had undergone and type of Depot Level Maintenance 
programs and what the effect was on reliability. Answers varied greatly. Some programs 
that had undergone DLM programs claimed drastic changes in reliability performance 
while others claimed there was no significant change following completion of the 
maintenance activities. Another respondent noted that his program was scheduled for 
mid-life rebuild, but the action was never carried out due to funding constraints. 

c. Reliability Growth Programs 

Reliability growth is the improvement in a reliability parameter over a 
period of time resulting from changes in product design or the manufacturing process. A 
structured reliability growth program is typically created with specific interim reliability 
goals and test events. As the system design matures, testing is performed at designated 


intervals to identify actual or potential sources of failure. 


Managing reliability growth requires systematic planning for reliability 
performance achievement as a function of time and other resources. This involves 
controlling the ongoing rate of achievement by reallocation of resources based on 
comparisons required, planned, estimated, and ass essed reliability values (Ryan, p. 42). 
Formal reliability growth programs serve to not only ascertain requirement compliance, 


but to also identify potential problems early in development. 


Survey participants were asked whether their programs incorporated a 


formal reliability growth program. With the exception of the AAAV program, still under 
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development, none of the programs examined had a formal reliability growth program in 
place. However, there was a general agreement that the focus should be on disc overing 
design flaws early and fixing them as early as possible to avoid potential cost overruns in 
the future of the program. 
H. CHAPTER SUMMARY 

This chapter presents the methodology used and the data gathered from the 
survey, interviews, and emails. Professionals within the Marine Corps acquisition 
community directly contributed by providing information about their programs, 
experiences, and perspectives with respect to reliability management. The data was 
organized into five major themes: 1) management approach to reliability, 2) determining 
and documenting reliability requirements, 3) contracting and incentivizing for reliability, 
4) reliability testing, and 5) comparing and assessing required, estimated, and achieved 


field reliability. 


It should be noted that the survey responses were from individuals that inherited 
the programs long after the systems were developed and fielded. Consequently, many of 
the responses were a result of a lack of documentation confirming actions taken during 
development and not necessarily a result of a lack of action on the part of the original PM 


staff. 


The next chapter provides an organized analysis of the data presented in this 
chapter. The author focuses on common inhibitors, enablers, issues, and risks associa ted 


with effective reliability management while discussing mitigation techniques. 
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V. PROGRAM RELIABILITY ANALYSIS AND LESSONS 
LEARNED 


A. INTRODUCTION 

This chapter provides an analysis and assessment of the common reliability 
management issues faced by Marine Corps Program Management Offices. The research 
results focus on the perspectives and opinions of acquisition personnel, as attained from 
the survey responses, and do not specifically address technology driven reliability 
problems. The analysis follows the format of the data presented in the previous chapter, 
organized around the five reliability management themes, developed for the purpose of 


this thesis, and listed below: 


e Management Approach to Reliability 
e Determining and Documenting Reliability Requirements 
e Contracting and Incentivizing for Reliability 


° Reliability Testing 
e Comparing and Assessing Required, Estimated, and Achieved Reliability 
While the research is limited to selected legacy principle end items, it is logical 
that many of the challenges, issues, and potential solutions correlate to other end items in 
the Marine Corps acquisition process or currently in operational use. 
B. ANALYSIS OF RELIABILITY MANAGEMENT ISSUES 
1. Management Approach to Reliability 
a. Factors Contributing to Reliability Performance Problems 
Weapon systems often fall short of the desired level of reliability planners 
and developers originally planned to achieve. While increased system complexity and 
harsh environmental and operational conditions likely contribute to the challenges 
associated with achieving reliability requirements, the researcher observes that a large 
portion of performance problems can be directly linked to management of reliability 
factors. The following survey responses cite the top five reliability issues in order merit 
ranking as viewed by acquisition professionals participating in this research: 


e Wrong Metrics - Traditional T&E RAM metrics are not supported by 
USMC maintenance data sources, and programs are unable to make a 
valid comparison between RAM requirements and estimates with achieved 
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field data. How can a program manage something that is not adequately 


measured? 

e Schedule Goals Outweigh Reliability — too much pressure to field systems 
rapidly 

e Need more qualified reliability personnel in Progr am Offices 

e Unrealistic Requirements - unrealistic reliability requirements with 


inadequate rationale; there seems to be a disconnect between user, 
materiel developer, and combat developer thoughts on_ realistic 
requirements 


e Poor Reliability Planning and Growth Planning — reliability growth is not 
properly utilized as a tool to reduce reliability related issues upfront 


b. Reliability Roles and Responsibilities 

Among those systems examined, there was no consistent managerial 
approach concerning who was delegated responsibility for reliability activities. However, 
all but one program cited at least one individual who was primarily responsible for 
reliability as survey respondents identified Logistics Team Leaders, Project Team 


Leaders, Supportability Team Leaders, or Reliability IPTs. 


As one former PM noted, PMs often manage by exception and without a 
specific problem or issue, reliability and the other engineering disciplines are managed 
through empowerment of technical experts. In this sense, it is interpreted that PMs 
depend upon outside reliability competency for matrix support. The reliability experts 
typically provide input on the specific reliability activities and where they should be 
implemented. How such an approach is incorporated into a program is dependent upon 
available funding, as well as the PM’s judgment based on cost, schedule, performance, 


and supportability considerations. 


Survey respondents identified the need for more qualified reliability 
personnel in program offices as the third top inhibitor to effective reliability management. 
By assigning a permanent reliability engineer or staff, just as the AAAV program has 
done, adequate technical expertise is available to the Program Manager. This would 
allow logistics engineers to partner with their counterparts in reliability engineering to 
collectively define and allocate reliability requirements affecting logistics. The duo must 


defend the logistics support concepts and supportability design requirements that they 
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propose, not only from the logistics community’s perspective, but also from the 


engineering point of view. 


The author believes that assigning an individual or team, such as a 
formally chartered reliability IPT, as the central authority on reliability activities ensures 
there is a reliability advocate that can defend related issues and identify concerns during 
the numerous trade-off analyses and discussions. 

Cc. Documenting a Program’s Reliability Approach 

The fifth ranked inhibitor to effective reliability management, as der ived 
from the survey, was poor reliability planning and growth planning. Consequently, half 
of the programs reviewed indicated that there was no formal reliability management plan 
while the other half simply relied on the contract SOW or TEMP to address the 
methodologies and plans for measuring and achieving reliability. Again, the exception 
was the AAAV program office, which uses a FRACAS database to formally document its 


reliability program plan. 


In order to provide visibility into the management and activities of those 
parties responsible (Government and contractor) for the reliability progress within a 
program, there should be definitive documentation on all reliability activities, functions, 
processes, test strategies, measurement/metrics, data collection, resources and timelines 
required to ensure reliability system maturation. Specifically, Reliability Program Plans 
(RPPs) can serve as a comprehensive document detailing all of the actions, functions, 
resources and timelines related to reliability. Such plans would especially prove 
invaluable if an initiative is implemented that would make predicted and demonstrated 
reliability a mandatory component of the acquisition world at each phase of the 
acquisition cycle. 

d. Tools Used in Evaluating Poten tial Failures 

Proactive reliability management early in the lifecycle of a system is 
typically more cost effective than coping with schedule delays and unanticipated costs of 
failing a test later, forcing costly redesign and additional testing to demonstrate that the 
problem is corrected. There are numerous test and design tools available to program 
offices and contractors that help to ensure reliability is “designed in” early in the 


program, but it is up to the program management to ensure such opportunities are 
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exploited. The risk mitigation techniques consist of testing, engineering, and other 
technical methods used to determine and evaluate potential system failures and their 
causes. However, the extent to which the legacy programs failed to utilize the reliability 
risk mitigation activities was surprising. Aside from developmental and operational 
testing, programs generally indicated use of only one of the following techniques: 

Environmental Stress Screening (ESS), reliability modeling, FMECA, Relia bility 
Development/Growth Test (RD/GT), or FRACAS. No (legacy) programs were aware of 
the use of reliability allocation, fault tree analysis, probabilistic failure assessment, 

reliability qualification test, PFMEA, Weibull analysis, physics of failure, or a parts 
control program. The AAAV program, currently under development, seems to be setting 
precedence for future weapons systems in terms of reliability management, as it was the 
only system examined which utilized all of the identified risk mitigation techniques 


identified by the survey. 


Many program representatives were either not aware of the specific 
techniques utilized to ensure reliability was “built-in,” or the original staffs did not use 
the available tools. However, there was a common consensus to test early and often, and 
use knowledge of reliability growth to implement corrective action. All PMOs reported 
using some form of failure analysis as an integral part of the design process, and there 
was a consensus that the use of such tools, which incorporate reliability prediction and 


achievement into system design, was beneficial. 


In most cases, the survey respondents were not the original PMO staff, and 
the respondents may have been aware of which techniques were utilized only by 
reviewing any existing documentation that was retained before their arrival. Often, 
documentation from the original staff or the contractor was no longer available, and thus, 
the assumption that the legacy systems did not take advantage of the reliability analysis 
tools may be invalid. In reality, many of the programs may have utilized the tools more 
than indicated in the survey, and the respondents were not aware of the previous staffs’ or 
contractors’ actions. 

e. Existing Policy, Guidance, and Regulations on Reliability 

The acquisition community either has little guidance or is not aware of 


guidance concerning reliability management. Survey responses indicate that the little 
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existing guidance is very broadly scoped, providing minimal detail as to specific 
reliability actions to be taken in the acquisition process. For example, the DoD 5000.2-R 
states that the “PM shall establish RAM activities early in the acquisition cycle.” 

Additionally, the amount of mandatory guidance is minimal and has further decreased in 
recent years, due to acquisition reform initiatives, and the applicability of discretionary 

guidance seems to be somewhat confusing to the acquisition community. Ironically, the 
DoD 5000 series was cancelled, while the author was collecting research for this thesis, 


because its guidance was deemed too restrictive in nature. 


The acquisition workforce, responsible for reliability activities, appears to 
need enforceable and precisely delineated criteria, standards, and procedures to guide it in 
the effective management of reliability. 

2. Determining and Documenting Reliability Requirements 

a. Influencing Realistic Reliability Requirements 

According to the survey responses, which indicate the materiel developer 
was able to provide input for establishing reliability requirements in nearly all programs, 
it appears that programs actively participate with the Combat Developer to determine the 
requirements relating to reliability. Ironically, the fourth top rated factor contributing to 
reliability management problems, as derived from the first survey question, was that 
PMOs were faced with “unrealistic reliability requirements with inadequate rationale.” 
This indicates that there is a disconnect between the user, materiel developer, and combat 


developer with regard to the perspectives of realistic requirements. 


The respondents indicated the opportunity is available to influence 
reliability requirements generation. It is essential that logistics and reliability expertise 
be involved in requirements generation, on behalf of the PM, in this early stage of 
program development. The extent to which the level of influence is effective is largely 


dependent upon their level of expertise and involvement. 


A review of the terms in which the reliability requirement is identifie d 
varies from program to program, indicating that there is not a standard operational 


terminology in which reliability must be expressed. While this likely allows for 
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flexibility, there must be an agreement and understanding between the Government and 
contractor of those terms, as examined in upcoming sections. 

b. Reliability as a KPP in the ORD 

While reliability requirements were not identified as KPPs, programs 
agreed that reliability was an important priority that received attention in the ORD. 
According to DoD 5000.2-R, reliability requirements are to address “mission reliability” 
and “logistics reliability,” implying that ORD requirements should focus on measures 
related to mission completion, such as operational availability. Appropriately, all of the 


systems examined had specific MTBOMF or MMBF requirements. 


The AAAV, which is still under development, is the only program that 
indicated the use of reliability as a KPP. In fact, the AAAV has a very specific 
MTBOMF threshold as a KPP for the Milestone C decision. To ensure that the 
requirement is in an objective and quantifiable term that the contractor and the 
Government can agree upon, according to the reliability survey responses, the AAAV 
contractor was “given the Failure Definition and Scoring Criteria which was the basis of 


determining whether a failure was an operational mission failure.” 


One assumption for the traditional systems not designating a definitive 
reliability KPP is that doing so would have compromised the flexibility and trade -off 
range available to PMs in a time where cost, schedule, and performance were the priority. 
However, it is generally agreed, in theory, that reliability and maintainability, along with 
performance, should act as equal partners in today’s requirements generation process. 
This is logical as reliability and maintainability contribute to combat power generation 
and are not severable from system performance. If reliability is not a KPP in the ORD, it 
simply gets pushed aside due to other requirements precedence. 

Cc. Reliability as a Source Selection Factor 

With the exception of the AAAV, the program respondents replied that 
either reliability was not a factor in source selection or they were not certain if reliability 
was a factor in source selection due to the time that had passed since the program was 
originally contracted and the lack of documentation in the PM offices. While reliability 
was not a source selection factor, some respondents gave the impression that, 


“Reliability, with its impact on O&S cos ts, should receive critical attention in the market 
96 


investigation, solicitation, and source selection process.” Reliability is a logical key 
source selection criterion to contribute to future equipment readiness improvements and 
to ensure adequate logistics weight in source selection. 

3. Contracting for Reliability 

The concept of reliability is often utilized without precise definition, while the 
terminology is non-standard throughout the logistics community and tends to depend on 
the system being developed. However, while creating DoD requirements documentation, 
contract specifications, and test documentation, it is very important that all main concepts 
are addressed in an unambiguous way so that all parties involved (to include the user, 


combat developer, materiel developer, PM, contractor, and tester) understand the terms. 


A common problem occurs during the translation of operational reliability 
requirements into contractual reliability requirements, as they typically are expressed in 
different terms. Operational reliability parameters, as derived by the user and Combat 
Developer, are often in terms of operational availability or mission duration needs. The 
Materiel Developer takes these parameters and allocates them to technical reliabilities of 
the system in the terms of MTBM, MTBOMF, or MTBF, traditionally the common 
parameters of reliability used for contractual purposes. The focus for the PM is to ensure 
the contractual reliability of the system, as measured in controlled test conditions, 
supports the dynamic and unpredictable environment in which operational availability is 
measured. The challenge for PMs remains in creating a definitive correlation between 
operational and contractual reliability, one that positively indicates anticipated system 


reliability performance. 


In order to achieve the reliability requirement specified in the ORD, additional 
levels of increased reliability may be added to the contract, just as the AAAV has done. 
This helps to account for the environmental and operational differences imposed during 
fleet operations. However, roughly two-thirds of the participating survey respondents 
indicated that the ORD paragraphs relating to reliability were simply restated in the 
Statement of Work or performance specifications, indicating that the contract 


requirement was very similar to the ORD requirement and open to interpretation. 
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The contractor and designer must consider the risks of field maintenance and 
minimize the characteristics of the design that are susceptible to operationally induced 
reliability deterioration. Likewise, contractor reliability predictions should be based on 
realistic operational projections for degradation. The operation and maintenance of 
equipment in the field can induce these effects by stressing sys tems beyond predicted 
levels. Operational contributors to such overstresses include neglect, unfamiliarity, 
carelessness, and mission constraints. Maintenance actions can also induce defects in 
otherwise satisfactory assemblies; foreign objects introduced, fasteners improperly 
engaged, contaminants introduced, improper part replacement, improper lubricants, etc. 
The contract provisions should attempt to account for all elements contributing to the 
combined failure rate and provide the Government with a confidence interval for a 
predetermined readiness performance in the form of operational availability. Ultimately, 
performance specifications should include an inherent reliability goal, and when such 
goals are not achieved, it should be considered a latent defect. 

4. Reliability Testing 

a. Resource Constraints 

A common perspective expressed by the program offices is a lack of time 
and money to conduct adequate levels of reliability testing, which are needed to achieve a 
substantial confidence level of the system reliability. In fact, data from the first survey 
question indicated that there is too much pressure to field systems quickly and too much 
pressure to field systems cheaply, which were, respectively, the second and ninth ranked 
inhibitors to effective reliability management. However, this is not surprising in the 
acquisition world where program offices continuously conduct trade-off analyses while 


competing for constrained resources in a politically affected environment. 


Funding and time constraints create trade-offs between reliability 
improvements and performance improvements. In reality, system performance 
improvements are often made at the expense of potential reliability enhancements. Such 
decreased emphasis on reliability resources typically increase life cycle costs as the 
acquisition community and those guiding it must acknowledge that funding not spent on 
reliability optimization during development will be spent multiple times over during 


operations and support. 
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Trading off reliability for another performance enhancement is not only 
costly in terms of Total Ownership Costs, but may actually make the system less combat 
effective as maintenance workload increases, adversely affecting system availability. 

b. Testing to Determine Reliability Performance 

A major focus for the PM is to ensure the contractual reliability of the 
system, as measured in controlled tests, is adequate for the dynamic and unpredictable 
operational environment in which the system is intended to operate. Without adequate 
Government testing, which is to provide a substantial confidence level of system 
reliability, DoD is often forced to utilize erroneous contractor or manufacturer estimates 
of reliability performance as a basis for logistics supportability decisions. Under- or 
over-estimating reliability will cause limited funds to be allocated unwisely. For 
example, inaccurate reliability estimates can potentially have devastating effects in terms 
of spare parts availability, the amount and level of mechanic training, repair facilities 
infrastructure, system operational availability, and increased life cycle costs. The 
contractor or manufacturer claims of reliability must be proven through independent 
testing, and claims should be considered unsubstantiated until tested. In other words, 
DoD must adopt the null hypothesis that the reliability is not what the contractor claims, 
but rather, what the contractor proves. To do so, the Government must validate the 
contractor’s estimates. Once the contractor submits their RAM estimate, it becomes the 
Government’s estimate only if they accept it. Therefore, it is up to DoD to become 
involved in this process and to conduct independent testing as necessary to verify such 
estimates. 

Cc. Political and Cultural Barriers Affecting Testing 

Program Managers are evaluated on procurement cost, schedule, and 
system performance, in accordance with Defense Acquisition Executive Summary, 
versus supportability, in the form of a target operational availability or life cycle cost. 
Consequently, PMs have traditionally had the incentive to reduce up-front procurement 
costs and field systems rapidly, often accomplished through reduced research, 
development, test, and evaluation. The effects of such incentives directly contribute to 
reduced reliability performance, decreased readiness, and increased life cycle costs. If 


the PMs, the prime contractor, and all members of the IPTs were evaluated on readiness 
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performance, and incentives were in place to reward for reliable and supportable systems, 
the effect would likely be an increased focus on RDT&E activities associated with 
reliability achievement, which would ultimately decrease life cycle costs while increasing 


operational availability. 


Commercial firms have tended to adopt more successful test evaluations 
because, as a whole, they have an appreciation for “why” testing is conducted vice “how” 
testing is conducted. In general, the culture of the commercial industry views testing as a 
method of discovering problems early which results in less expenses later. Corporate 
support for new product development defuses test results as a threat to program support. 
Conversely, it is difficult for weapon system programs to compete for approval unless 
they offer significantly better performance over competing systems while conforming to 
funding and schedule constraints. In this sense, test results tend to become scorecards 
that indicate whether a program is ready to proceed or to receive the next increment of 
funding, an activity that is seemingly intended more for the decision makers above the 
program. Thus, program managers have incentives to postpone difficult tests while 
minimizing open communication about test results (GAO, “A More Constructive Test 


Approach . . ., p. 8). 


An initiative to make reliability estimates and reliability achievement 
demonstration a mandatory part of each phase of the acquisition cycle would alleviate the 
incentives for program managers to postpone difficult testing. If implemented, the 
initiative would help ensure the attainment of increasing levels of system reliability as the 
program matured, by requiring demonstrated levels of reliability before major 
programmatic approvals and milestones. 


5. Comparing and Assessing Required, Estimated, and Achieved 
Reliability 


It is important to have a systematic process in place for collecting and comparing 
reliability data for several reasons. The Marine Corps must be able to calculate and 
compare the reliability that is being achieved in the field during post-production with the 
required and estimated reliability in order to determine contactor compliance, 
successfully hold contractors to their estimates, and determine if the user reliability 


requirement is being met. 
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The basic policy of DoD is to hold contractors responsible for quality of the 
products through various types of quality assurance programs. This obviously requires a 
plan and action, which must be based on the quality requirements outlined in the ORD. 
To do so, it is recommended that a program use the reliability requirements stated in 
operational requirements, or those resulting from trade-off analysis, as a baseline for 
reliability assessment to be compared with actual achieved field reliability. However, the 
difficulty remains in collecting, interpreting, and comparing operational (achieved) 
reliability with contractual reliability measurements. Aside from the essential collection 
of achieved field data, original contractor estimates and ORD requirements must be 
retained for comparison, and sustainment organizations must have the resources to 


enforce contractual terms. 


It is well known that “you can’t manage what you don’t measure,” and in general, 
survey responses indicate that there is a lack of a systematic process for collecting 
reliability trend data beyond readiness ratings for Marine Corps ground equipment. 
Furthermore, what data that does exist is suspect to error and corruption as a result of the 
current maintenance management automated information systems. There was also a 
consensus that traditional test and evaluation RAM metrics are not supported by 
maintenance management data systems. This was voted as the top rated inhibitor to 
effective reliability management, according to the responses from the first question of the 
survey. With these factors in mind, it is seemingly impossible to compare contractual 
reliability and/or test estimates in the form of MTBF with corrupted operational 
reliability data in the form of MTBM, as attained from the Marine Corps maintenance 
information systems. 

a. Maintaining Reliability Requirements and Reliability Estimates 

Documentation of contractor estimates is not typically retained while 
ORD reliability requirements are often difficult to locate. None of the participating 
programs were able to identify the level to which the contractors’ estimates were 
demonstrated (during testing phases or sustainment) due to the fact that contractor 
estimates were not retained. In some cases, the ORD could not be found, meaning the 
original reliability requirement was not retained. These facts indicate that there is an 
apparent traceability issue within the program offices of legacy systems. 
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b. Calculating Achieved Field Reliability 

Most of the programs surveyed indicated that achieved field reliability 
data compiled during the sustain ment phases of the respective systems is suspect to error, 
making the comparison of such data with the contractor estimates and original ORD 


requirements (when available) questionable. 


When attempting to compare reliability requirements, reliability estim ates, 
and achieved reliability, there are numerous data deficiencies that the Marine Corps has 
been forced to overcome. First, the calculation of MTBF is unfeasible at this time 
utilizing the traditional maintenance data available from current maintenance 
management systems. Next, it is arguable whether MTBM is a feasible surrogate for 
MTBF, due to the inclusion of preventive maintenance actions used in calculating this 
measurement. Even so, the data used for MTBM calculation is often skewed for various 
reasons, discussed in following sections. Thus, the logistics and program management 
communities have been forced to use the “next best measurement,” operational 
availability, to attempt to measure system reliability. The question remains whether 
operational availability can be used or translated to make a valid comparison with the 


contractors’ original reliability estimates in terms of MTBF, MTBOMF, MTBM, etc. 


Mean Time Between Failure. It is important to distinguish why MTBF 
needs to be calculated for equipment. First, contractor reliability estimates are typically 
provided in the form of MTBF, as derived from operational requirements. As a result, 
the contractual reliability, expressed in inherent values and used to define, measure, and 
evaluate the contractor’s program, is also typically in the form of MTBF. In order to 
successfully hold contractors to their estimates, the Marine Corps must be able to 
calculate the reliability that is being achieved in the field during post-production. 
Second, the calculation of this time is also necessary in order to determine whether the 
mean time between failures is increasing, decreasing, or remaining constant with age. As 
equipment ages, its MTBF decreases until the cost of keeping that item operational is 
more than the cost of buying a new item. Estimates of when maintenance costs will 
exceed acquisition costs are questionable without mean time between failure calculations 
(Enholm, p. 1). In other words, MTBF data analysis helps to determine or confirm if 


equipment is in the “wear-out” phase of its life cycle and at the end of its economic 
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useful life. Next, the determination of MTBF can serve to indicate whether Depot Level 
Maintenance programs are providing a cost effective benefit by comparing reliability 
metrics prior to, and after depot level maintenance, allowing decision makers to tradeoff 
expected readiness for DLM cost avoidance. Lastly, MTBF could be used as an input to 
determine the optimal provisioning of spare parts, utilizing commercial Readiness Based 


Sparing (RBS) packages and techniques. 


Operational usage data is required to calculate MTBF. However, the 
current Marine Corps maintenance management data systems are not capable of tracking 
operational usage. Additionally, there are concerns as to which time unit of measurement 
is most appropriate for calculating MTBF. While MTBF is traditionally calculated as a 
time between failure, other units such as mean mileage between failures, mean operations 
or starts between failures, or mean rounds fired between failures may be better indicators 
of failure intervals. It may be possible to construct a sophisticated algorithm to determine 
a mean mileage (or other form of operation) for specific fleet equipment, but the error 


rate would be relatively high due to the often inaccurate and unusable meter readings of 


USMC field equipment. 


For systematic failure or maintenance degradation, there is also a 
requirement that the age of the system be established. Particularly, this is important when 
attemptin g to establish the age when a system starts spending more time in a non-mission 
capable status as it gets older and maintenance requirements start adversely affecting 
operations. However, there is not a central repository or data source where the Marine 
Corps collects information establishing the economic useful life of serialized items. 
While the program management offices sometimes keep a logbook detailing when 
specific serial numbers were fielded, the information is not always readily available 
(Enholm, p. 2). 

MTBF vs. MTBM vs. A,. The Marine Corps is forced to substitute MTBF 
with either MTBM or A, due to lack of operational usage data needed to calculate 
MTBF. The feasibility of this substitution is questionable due to the inclusion of both 


preventive and corrective maintenance actions in the calculation of MTBM. 


Additionally, even the calculation of MTBM is often suspect to error due to various 
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factors contributing to a lack of quality maintenance data, and as indicated in the previous 
chapter as well as Appendix C, MTBM can only intermittently be calculated for certain 


equipment. 


As a result, “R-rating” or readiness rating is computed per Marine Corps 
Bulletin (MCBul) 3000 as a substitute to both MTBF and MTBM, offering what is 
perhaps, the most feasible estimate of a measurement for reliability achievement. R- 
rating basically provides a snapshot of operational availability (A,), as discussed in 


Chapter II. Recalling that, 


UPTIME 
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it is important to note that the Administrative Logistics Delay Time (ALDT), Corrective 
Maintenance Time (CMT), and Preventive Maintenance Time (PMT) are included in the 
calculation of downtime when computing operational availability. This is largely due to 
the fact that such variables are difficult to extract and distinguish amongst one another as 
a result of weaknesses with the current maintenance management data systems and 
tracking procedures. Operational availability can also be improved by maintaining a 
large stockage of frequently used spare parts. While availability is improved, the 


logistics burden is significant. 


Factors Contributing to Skewed Data. As indicated by a majority of the 
survey respondents, there is an obvious lack of quality historical maintenance data, 


attributable to numerous causes, some of which are highlighted in this section. 


When attempting to compare the required, estimated, and achieved 
reliability of weapon systems with the limited data available, it is important to take into 
account that the respective readiness rates most often include all inventory within the 
Marine Corps. This may actually skew the snapshot of overall achieved reliability for 
particular systems. In other words, reliability based on readiness rates may appear to be 
higher than what is really being achieved due to the equipment in stores and on Maritime 
Pre-positioning Force (MPF) ships positively affecting the overall readiness rates for a 
specific system. The stores and MPF equipment have an extremely low usage rate and 
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are well maintained. An additional data set that excludes stores and MPF equipment 


would be required to test this hypothesis. 


Depot Level Maintenance Programs are conducted on systems that have 
shown trends of decreasing readiness corresponding with increased age, that have 
exceeded or approached their expected useful lives, or that are will be required to be 
operational for extended periods without a replacement system. DLM programs are 
specifically designed to have a significant improvement on the reliability, availabilit y, 
and maintainability of the systems. As a result, there is likely an impact on the achieved 
reliability data that is collected for comparison to required and estimated reliability. In 
order to accurately account for the effects of improved reliability, the data collected on 
the systems must be separated into those that have undergone DLM and those that have 


not. 


Selective interchange, replacement of major system components, and 
maintenance actions, in general, affect the overall inherent reliability that a system was 


” 


“designed to.” As such actions change the reliability and availability of an end item, it 
becomes virtually impossible to compare what the system was supposed to achieve, 
according to contractor/Government estimates, and what it is actually achieving in the 
field. Additionally, “operator- and maintainer-induced failures” result in a reliability 
achievement that is less than the inherent reliability of the system, but these incidents are 


also largely unknown. 


Lastly, Equipment Repair Orders (EROs), used to initiate maintenance 
activities, often include multiple failures on a single document, creating inconsistencies 
in the calculation of MTBM. Also, the source of the data maintenance and supply data is 
from two disparate systems, MIMMS and ATLASS, respectively. 

C. “Reliability Gap” 

The absence of retained data within the acquisition and logistics 
communities, combined with the lack of survey participation, led to an inconclusive 
hypothesis whether a “reliability gap” actually exists between the original reliability 
requirements, contractor/Government estimates, and achieved field reliability. However, 


the survey respondents’ acknowledgements to an absence and inaccuracy of the requested 
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reliability data supports the concept that there is an issue with respect to calculating, 
retaining, comparing, and assessing reliability achievement and trends of mature systems 
as well as original contractor reliability estimates and ORD requirements. In this case, 
“no data, is data.” 
C. LESSONS LEARNED 

Program Offices of Marine Corps legacy weapon systems, procured decades ago, 
had not always taken advantage of effective reliability management opportunities. The 
proper level of managerial attention, in the realm of reliability, was not given to the 
systems primarily due to the lack of focus on RAM metrics and LCC concerns. 
However, it is promising to see the increased focus and attention given to reliability 
performance of the AAAV program, likely representative of future programs to be 


developed within Marine Corps acquisitions. 


Today, it is well known that there are many opportunities to address reliability 
within weapon systems acquisitions - from requirements generation and contracting to the 
conceptualization, design, and development utilizing the systems engineering process to 
demonstration and incremental testing throughout development to the operational 
monitoring and comparing of achieved reliability with the estimated reliability to ensure 
attainment of reliability as planned. However, procedural, managerial, and incentive 
pressures still force program offices to sacrifice reliability for the achievement of other 


goals such as time, money, and performance. 


The concept of reliability is often utilized without precise definition, while the 
terminology is non-standard throughout the logistics community and tends to depend on 
the system being developed. However, while creating DoD requirements documentation, 
contract specifications, and test documentation, it is very important that all main concepts 
are addressed in an unambiguous way so that all parties involved (to include the user, 
combat developer, materiel developer, PM, contractor, and tester) understand the terms. 
A common problem emerges in the translation of operational reliability requirements into 
contractual reliability requirements, as they typically are expressed in different terms. 
Operational reliability parameters, as described by the user and Combat Developer, are 
often in terms of operational availability or mission duration needs. The Materiel 


Developer takes these parameters and allocates them to technical reliabilities of the 
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system in the terms of MTBF, traditionally the common parameter of reliability used for 
contractual purposes. Then, the focus for the PM is to ensure the contractual reliability of 
the system, as measured in controlled test conditions, supports the dynamic and 


unpredictable environment in which operational availability is measured. 


The essential documentation and collection of achieved field teliability data, 
original contractor estimates, and reliability requirements are not typically retained or are 
difficult to locate, making comparison of such measurements impossible. In fact, none of 
the participating programs were able to identify the level to which the contractors’ 
estimates were achieved during operational testing, developmental testing, and the 
sustainment phase due to the fact that contractor estimates were not retained by any of the 
programs examined. In some cases, the ORD could not be found, meaning the 
Government’s original reliability requirement was not retained either. Additionally, 
many of the survey participants noted that developmental test data was difficult to come 
by while operational test data was somewhat more readily available due to the role of 
independent Government testing agencies, such as MCOTEA. Lastly, a difficulty 
remains in collecting and computing accurate operational reliability data as most 
programs indicated that achieved field reliability data compiled during sustainment of the 
respective systems is suspect, making the comparison of such data with the original ORD 


requirement and contractor estimate (when available) questionable. 


Prior to research and data collection attempts, PMOs seemed to be the likely place 
to find original documents and data such as ORDs, reliability requirements, contracts, 
contractor reliability estimates, and achieved reliability data. Research ascertained that 
there was a retention, traceability, and computation problems with original documents 


and reliability data. 


As demonstrated in the MTBF/Maintenance Study (Appendix C) conducted at 
MARCORMATCOM, it is possible to calculate a sample mean or median time between 
failure (and maintenance combined) for some Marine Corps equipment using advanced 
methodologies, but the accuracy of the calculations is suspect due to weaknesses in 


maintenance management data systems. Likely due to the difficulty, inability, and errors 
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involved with such attempts, there is not a requirement to track the MTBM or MTBF of 


systems during their operational life cycles. 


The maintenance data available using the Marine Corps’ current maintenance 
management data systems (MIMMS/ATLASS II/SASSY) does not provide the adequate 
information necessary to calculate MTBF and is suspect to corruption and error. 
Specifically, the Marine Corps is unable to calculate failure rates because there is not a 
way to track the operational usage of end-items. Perhaps a feasible solution will arise 
with the implementation of the Global Combat Support System -Marine Corps (GCSS - 
MC). The GCSS-MC system will replace our current legacy maintenance systems and 
could possibly contain serialized records for primary end items, permitting the tracking of 
operational usage, which is necessary in the calculation of MTBF. Navy and Marine 
Corps supply/maintenance procedures used to track flight hours on naval aircraft may 


offer possible solutions to tracing operational usage of USMC ground equipment. 


Managing reliability does not end with OT&E and fielding of the system, and 
instead, reliability must be continually monitored and assessed for potential 
improvements and efficiencies in support of meeting Marine Corps life cycle cost and 
readiness objectives. In fact, once a system is fielded, reliability assessment should 
become a permanent part of sustainment activities conducted by Program Management 
Offices as well as other Life Cycle Management organizations. To be successful, 
reliability growth must continue during the customer-use phase by coordinating feedback 
from the warfighters to the suppliers and by supporting necessary corrective actions. 

D. CHAPTER SUMMARY 

This chapter analyzed common PM issues and challenges involved with managing 
the reliability performance of Marine Corps grou nd equipment. The analysis was based 
on program data and results from a reliability management survey administered to 
various personnel associated with the acquisition process. The chapter was formatted 
around the five major reliability management issues, derived for the purpose of this 
research. The final chapter will provide selected conclusions focused on the research 
questions while relaying some recommendations on how to best approach reliability 


issues from a management standpoint. 
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VI. CONCLUSIONS AND RECOMMENDATIONS 


A. INTRODUCTION 

Research conducted in support of this thesis evaluated system reliability 
management in the acquisition process as it applied to selected principle end items within 
the Marine Corps ground equipment inventory. Common inhibitors and enablers of 
effective reliability management, why they occur, lessons learned, and potential methods 
for mitigating the inherent risks are collectively summarized and analyzed as derived 
from surveys, interviews, and existing maintenance data. The research ascertains a 
variety of technical, programmatic, managerial, incentive, and procedural issues that the 
Marine Corps encounters concerning system reliability requirements and achievement. 
The overall intent of this research is to provide P rogram Management Offices, acquisition 
organizations, and strategic planners insight into the collective experiences and 
perspectives of acquisition workforce professionals who are familiar with issues relating 
to reliability management. The results of the thesis serve to provide insight into the 
improved sustainability of future systems while encouraging the development of a 
mechanism that enables the traceability of contractual reliability performance 


requirements to operational reliability requirements. 


In this final chapter, as a result of feedback and analysis of survey responses, 
selected conclusions are presented that focus on and answer the research questions, while 
potential recommendations are identified on how to best approach reliability issues from 
a managerial standpoint. The thesis concludes with recommended areas for further 
research. 

B. CONCLUSIONS AND RECOMMENDATIONS 

Subtle flaws in design affect system reliability and can have multi-million dollar 
effects as LCC continue to escalate throughout the life of the system. In the extreme, 
such subtle design flaws can become potentially catastrophic. The DoD needs to focus 
on the goal of providing a system that maximizes its operational availability within the 
targeted life-cycle cost of the program, and one of the primary methods of doing so is to 


practice effective reliability management. 
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1. Documentation and Data Retention Required for Reliability 
Assessment 


Conclusion: Prior to research and data collection attempts, PMOs appeared to be 
the likely place to find original documents and data such as ORDs, reliability 
requirements, contracts, contractor reliability estimates, and achieved reliability data. 
The failure to retain such documentation and the absence of reliability data within the 
acquisition and logistics communities, combined with the lack of survey participation, led 
to an inconclusive hypothesis whether a “reliability gap” actually exists between the 
original reliability requirements, contractor/Government estimates, and achieved field 


reliability. 


However, research ascertained that there was a retention and traceability issue 
with reliability requirements, reliability estimates, and original documents. The survey 
responses, citing an absence and inaccuracy of the requested reliability data, indicate that 
there is an issue with respect to calculating, comparing, and assessing reliability 
achievement and trends of mature systems as well as retaining the original contractor 
reliability estimates and ORD requirements. In this case, the author believes that the lack 


of this data was central to the intent of the thesis. 


Recommendation: In order to hold contractors to their original reliability 
estimates, it is recommended that a baseline for reliability assessment, such as_ the 
reliability requirements stated in operational requirements and _ performance 
specifications, be maintained to be compared with achieved field reliability. The well - 
meaning, but ineffectual philosophy often applied to reliability — “we will do the best we 
can” must be replaced with a contractual obligation in the form of measurable 
quantitative system reliability requirements and appropriate databases of fielded system 
reliability performance. 

2. Reliability Management Control Systems 

Conclusion: The Marine Corps acquisition community lacks adequate 
management control systems to ensure reliability performance meets predetermined 


requirements and contractor pre-fielding estimates. 


Recommendation: | Mandate that required, predicted, and demonstrated 


reliability be made a mandatory component of each phase of the acquisition cycle; that is, 
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require that realistic reliability predictions and demonstrations of achievement be 
incorporated into each Milestone Decision to be compared with the original requirements. 
This would allow the MDA to address reliability performance progress and plans for 
achieving the reliability thresholds of a system at every major review of a program. To 
do so, thresholds, or intermediate benchmarks representing minimum reliability 
achievement levels, should be established at various points of the program as a risk 
management technique. A breach of such a threshold could serve to indicate that the 
program is not on track in terms of reliability requirements, and intervention may be 


required to correct the discrepancy, if necessary. 


The Acquisition Program Baseline should continue to serve as a “contract” of 
sorts between the PM and the MDA. Reliability related parameters such as MTBF, Ao, 
and MTBM must exist for each program either in the Performance or Supportability 
sections of the APB. The acquisition program baseline status of each program must be 


reviewed at regular intervals and at major reviews. 


Additionally, when a program reaches a major milestone or experiences a 
significant change in its program parameters, the outcome is documented in an ADM. 
The Acquisition Decision Memorandum typically includes additional directives and 
statements with which the PM must comply, and thus, it needs to be approached as an 
opportunity for the MDA to encourage the achievement or improvement of reliability 
levels, while placing entrance criteria, constraints, or follow-on actions related to 
reliability on the programs. 

3. Reliability Policy, Regulations, and Guidance 

Conclusion: Predominantly as a result of acquisition reform, acquisition 
workforce professionals lack adequate guidance and requirements in the form of policy, 
publications, and directives, to guide them in the conduct of reliability activities. 
Additionally, the amount of mandatory guidance is minimal and has further decreased in 
recent years due to acquisition reform initiatives. Consequently, the acquisition 
community often utilizes an abundance of cancelled MIL-STDs and MIL-HDBKs or 


vague discretionary guidance to guide them in reliability related decision-making. 
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Recommendation: DoD should couple with commercial industry prime 
contractors to develop a comprehensive set of reliability management standards, which 
consider life cycle cost and sustainment consequences of early life cycle decisions. 

4, Reliability Roles 

Conclusion: Throughout the program offices of the legacy systems examined, 
there was no consistent managerial approach to reliability responsibilities, and while this 
may allow for flexibility, programs sometimes seemed to lack a clear understanding of 


who is responsible for reliability activities within a program. 


Recommendation: There should be a formally chartered Reliability IPT, 
responsible for ensuring effective communication between the program, us er, contractor, 
RAM community, engineers, testers, and logisticians. The IPT must be involved from 
concept exploration, affecting requirements establishment, analysis, and influence over 
the design factors. 

5. Reliability Requirements Generation 

Conclusion: There does not seem to be a consistent process for establishing 
operational reliability requirements performance measures during attempts to “link 
reliability performance to mission or supportability measures”, as required by DoD 
5000.2-R. In other words, the terms in which the reliability requirement was identified 
varied from program to program. Additionally, reliability has typically not been a Key 
Performance Parameter in the Operational Requirements Document for Marine Corps 
legacy systems, and, it gets pushed aside due to other requirements precedence to include 
cost, schedule, and performance objectives. Lastly, amongst the legacy systems 


examined, reliability was not used as a source selection factor. 


Recommendation: It is recommended that standards be established for defining 
reliability measures in the ORD. Reliability, along with cost, maintainability, and 
performance, should be considered equally in the requirements generation process, a 
stage at which the Materiel Developer and Combat Developer should be jointly defining 
realistic, achievable reliability requirements. To attain desired reliability performance 
thresholds and goals, it is recommended that robust reliability requirements be clearly 
defined and communicated as KPPs in the ORD, in clear operational terms by the 


Combat Developer. Likewise, reliability should appropriately be considered as a source 
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selection factor during solicitation. The reliability objectives must then be translated into 
quantifiable and verifiable contractual terms, traceable back to the system performance 
and the operational requirements. The Materiel Developer must adequately translate the 
system operational terms into viable contractual terms, understood by all parties involved 
to include the user, the program office, and the contractor so that compliance can be 
adequately monitored and enforced. Ultimately, it is recommended that Performance 
Specifications include an inherent reliability goal, and when such goals are not achieved, 
it should be considered a latent defect. 

6. Reliability Program Plan 

Conclusion: For the most part, Marine Corps legacy programs did not have 
structured reliability management processes in place, nor did they have corresponding 
overarching documents that define the activities, schedules, resources, and reliability 
achievement strategies needed to provide managerial insight into the programs’ reliability 
objectives. Consequently, Program Management Offices have not traditionally taken 
advantage of the numerous test and design tools available to them and contractors that 
help to ensure reliability is “designed in” early in the program by determining potential 
failures and the causes of such failures. “Designing-in” reliability upfront reduces risk 
and is less costly than finding design discrepancies during later stages of testing, 


evaluation, and operational use. 


Recommendation: In order to provide visibility into the management, functions, 
and responsibilities of those parties responsible (Government and contractor) for the 
reliability activities within a program, require all PMs to develop a Reliability Program 
Plan. This should be a mandatory document for all Milestone Decision Reviews, 
providing definitive documentation on all reliability activities, functions, processes, test 
strategies, measurements/metrics, data collection, resources and timelines required to 
ensure system reliability maturation. 


7. Impact of Inaccurate Reliability Estimates: Tying Contractors to 
their Estimates 


Conclusion: Foremost, systems that fail to meet reliability goals do not perform 
as expected, and the failure to meet such performance expectations proves to be costly in 


both operational and financial terms. Logistical support decisions are based upon 
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expected system reliability as determined early during development. Consequently, 
inaccurate reliability estimates significantly increase life cycle costs while adversely 
affecting the quality of logistical support available throughout the systems’ life cycles. In 
other words, when achieved field reliability is significantly less than what was 
anticipated, numerous deficiencies occur in the attempt to support maintenance 
requirements to include inadequate initial provisioning of spare parts, insufficient number 
and level of training for mechanics, deficient facilities and test equipment, and inadequate 


funding plans for future budgets. 


Recommendation: Contractors must be tied to LCC through their reliability 
estimates. Attempts must be made to use reliability incentives and warranties, and the 
Marine Corps must establish a mechanism that allows for the traceability of contractual 
reliability performance requirements to operational performance requirements. Likewise, 
the DoD must adopt the null hypothesis that the reliability is not what the contractor 
claims, but rather, what the contractor proves. To do so, the Government must validate 
the contractor’s estimates. Once the contractor submits their RAM estimate, it becomes 
the Government’s estimate only if they accept it. Therefore, it’s up to DoD to become 
involved in this process and to conduct independent testing as necessary to verify such 


estimates. 


Another alternative is to utilize Contractor Logistical Support (CLS) for a 
specified interim period to ensure contractors are tied to acc urate reliability estimates 
prior to transitioning the logistical support role to the military. After an accurate 
operational reliability measurement is determined through field operations, the Marine 
Corps can enforce contract provisions and optimally plan for and implement logistics 
support based upon actual achieved reliability. Traditionally, CLS has been utilized in 
cases where the military was not yet in a position to provide logistical support for a 
system that needed to be fielded rapidly. The author proposes the intentional use of 
planned CLS for a predetermined period, perhaps two to three years, to ensure that the 
system reliability is what the contractor had claimed, to be able to enforce contract 
provisions, and to plan for effective system supportability to achieve desired system 


performance. 
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8. Reliability Assessment Metrics 

Conclusion: Traditional T&E RAM metrics are not supported by current Marine 
Corps maintenance data sources, and thus, it is difficult to make a valid comparison 
between reliability requirements, as stated in the ORD; contractor RAM estimates; and 
data from actual achieved reliability of fielded systems. Specifically, USMC 
maintenance management systems do not track the necessary operational usage data 
required to accurately compute failure rate (in the form of MTBF), and instead, the 
current systems enable the computation of maintenance rate (in the form of MTBM), 
which includes both preventive and corrective maintenance as well as other skewing 
factors. The accuracy of the MTBM calculation is suspect to corruption and error due to 
weaknesses in maintenance management data systems and inefficiencies in the 
administrative maintenance processes. As a result, it is extremely difficult to accurately 
compare contractual reliability and/or test estimates in the form of MTBF with corrupted 
operational reliability data in the form of MTBM, as attained from the Marine Corps 


maintenance information systems. 


Additionally, the concept of reliability is often utilized without precis e definition, 
and the terminology is non-standard throughout the logistics community, tending to 
depend on the system being developed. However, while creating DoD requirements 
documentation, contract specifications, and test documentation, it is very important that 
all performance terms, including supportability performance terms, are addressed in an 
unambiguous way so that all parties involved (to include the user, combat developer, 
materiel developer, PM, contractor, and tester) understand them. A common problem 
emerges in the translation of operational reliability requirements into contractual 
reliability requirements, as they typically are expressed in different terms. The focus for 
the PM is to ensure the contractual inherent reliability of the system, as measured in 
controlled test conditions, supports the dynamic and unpredictable environment in which 


operational availability is measured. 


Recommendation: | Foremost, the operational and contractual reliability 
requirements must be measurable, verifiable, and most importantly, they must be 
comparable in an objective and quantifiable form that are contractually enforceable and 


that contractors and the Government can easily agree upon. 
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The author recommends that a feasible solution to track operational usage, 
enabling the calculation of MTBF, be included with the implementation of the Global 
Combat Support System-Marine Corps (GCSS-MC). The GCSS-MC system will replace 
current USMC legacy maintenance systems, and the author suggests that the replacem ent 
system contains serialized records for primary end items, permitting the tracking of 
operational usage, which is necessary in the calculation of MTBF. Additionally, weapon 
systems should be designed that, through modern technology, are able to self -monitor 
operational usage in terms of hours, starts, rounds fired, miles, or other applicable 
metrics. Another alternative is to consider Navy and Marine Corps supply/maintenance 
procedures used to track flight hours on naval aircraft as ways to offer possible solutions 
to tracing operational usage of USMC ground equipment. 

9. Cost Metrics 

Conclusion: Under current methods of utilizing unit production cost as a metric 
that determines the success of a program for the MDA, reliability and life cycle cost 
issues are often ignored to produce a lower unit cost. Likewise, while the acquisition 
workforce recognizes the significance of reliability, it typically gets pushed aside in the 
short-term crisis management environment of a constrained resources acquisition 


community where PMs are evaluated on procurement cost, schedule, and performance. 


Recommendation: Mandate the use of life cycle cost, within performance 
parameters, as the basis for all design trade-offs. However, DoD must develop a 
performance measure and incentive structure that recognizes life cycle cost equal to, or 


higher than the current acquisition cost, schedule, and performance metrics. 


Additionally, there needs to be a greater awareness by program management 
offices of the availability and capabilities of commercial LCC models. Such models 
should be utilized to assess the reliability, LCC, and logistics supportability impacts of 
various equipment configurations and other design and supportability issues. The 
commercial models should be used to provide design and support tradeoff, with 
sensitivity and comparative analysis, to ensure the program meets adequate reliability 


goals within, or below respective LCC budgets. 
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10. Inherent vs. Achieved Reliability: Consideration of Reliability 
Degradation 


Conclusion: There will likely be a difference between inherent (or potential) 
reliability and achieved reliability as demonstrated in the field. The operation and 
maintenance of equipment in the field often induces these effects by overstressing 
systems beyond predicted levels as a result of neglect, unfamiliarity, carelessness, and 
mission constraints. Additionally, the true achieved reliability can be obscured by 
scheduled and unscheduled maintenance actions and the corresponding administrative 
actions, conducted by inadequately trained personnel, mandated by incorrect diagnosis, 


or simply poorly managed. 


Recommendation: While a major effort is made in operations to reduce the 
effects of reliability degradation caused by maintenance, the desig ner should consider the 
risks of field maintenance and minimize the characteristics of the design that are 
susceptible to operationally induced reliability deterioration. Every effort should be 
made by both the Materiel Developer and the contractor (throu gh contractual language) 
to minimize potential human error. This may be accomplished with technologies such as 
bit/bite, diagnostics, prognostics, and autonomics. Equally important, reliability 


predictions should be made on realistic operational projections for degradation. 


Contractors should account for all elements contributing to the combined failure 
rate and provide the Government with a confidence interval for a predetermined readiness 
performance in the form of operational availability. Such concepts remain the premise 
behind the idea of Performance Based Logistics, which is a strategic approach to provide 
long-term measurable product sustainment to the warfighter. A realistic reliability 
requirement must account for all application environments and the time proportions 
expected in each phase or product life cycle of a system such as storage, transportation, 
installation, standby, and operation, and an apportionment of the requirement across the 
life cycle phases must account for deterioration in each phase. 

11. Continuous Reliability Assessment 

Conclusion: Managing reliability cannot end with OT&E and fielding of the 
system. Typically, legacy weapons systems did not implement formal reliability growth 


plans to ensure established reliability maturity levels were achieved during system 
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sustainment periods. Performance thresholds linked to system sustainability may or may 


not have been met, and unfortunately, the USMC does not know which is the case. 


Recommendation: Reliability must be continually monitored and assessed for 
potential improvements and efficiencies in support of meeting Marine Corps life cycle 
cost and readiness objectives. Once a system is fielded, formal reliability assessment 
should become a permanent part of sustainment activities conducted by Program 
Management Offices under the realm of the life cycle Weapon System Manager with 
assistance from the Logistics Management Specialist. To be successful, reliability 
growth must continue during the customer-use phase by coordinating feedback from the 
warfighters to the suppliers and by supporting necessary corrective actions. The data 
required for this effort must be collected in a method that is transparent to the operators 
and maintainers. 

C. RECOMMENDATIONS FOR FURTHER STUDY 

The following topics could have significant influence in the area of reliability 
management but fall beyond the scope of the present research, and thus, they are 
recommended topics for additional research: 


e Conduct a quantitative assessment to determine the level of confidence 
that can be realized when substituting MTBF with MTBM or A, 
calculations. Such a study could help to more accurately determine if 
these measurements are feasible surrogates for one another. 


e Conduct a similar study that encompasses a larger spectrum of programs, 
to include various ACAT levels as well as systems in various acquisition 
phases. 

e Conduct a similar survey conducted from the perspective of the contractor, 


who is expected to provide systems capable of achieving a predetermined 
required level of reliability. 


e Examine the feasibility of utilizing Contractor Logistical Support for a 
specified interim period to ensure contractors are tied to accurate 
reliability estimates prior to passing the logistical support role onto the 
military once actual reliability is determined through field operations. 


e Analyze the applicability and feasibility of utilizing reliability warranties 
and incentives in contracts to achieve more accurate reliability estimates 
from contractors to which they can be help accountable. 


e Using a commonly identified commercial life cycle cost model, conduct a 
sensitivity analysis of how changes in reliability affect operations and 
support costs. 
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e Conduct a study to determine the political and cultural barriers to 
increasing RDT&E investment, a funding investment that although 
increases procurement costs, may lead to more accurate reliability 
estimates and ultimately decrease LCC. 


D. THESIS SUMMARY 

Weapon system reliability demands constant managerial attention and 
implementation of effective management and technical strategies that balance cost, 
schedule, and performance with reliability during systems’ entire life cycles — from 
conceptualization and development to fielding and operational support through disposal. 
It is mperative to have early identification of upfront cost-effective opportunities for 
achieving the required reliability while obtaining and utilizing accurate reliability 
estimates for logistical support decisions. Mitigation of associated reliability risks during 
design, manufacturing, development, testing, and post-production operations must be 
accomplished to reduce the potential for unexpected life cycle cost inflation and 
decreased operational availability. Likewise, programs that have carefully planned and 
executed reliability management techniques will be more combat effective, in the way of 
decreased life cycle costs and increased operational availability, during the useful life of 


the systems. 


The bottom line remains that increased reliability of weapon systems contributes 
directly to greater combat effectiveness. To fight and win wars, Marines must be 
equipped with systems that are reliable. Progress must continue to be made to ensure that 
inhibitors to effectively provide such lethal systems to the fleet are recognized and 


mitigated. 
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APPENDIX A. ACQUISITION RELATED ACRONYM S 


ADM 
ACAT 

Aa 

Aj 

ALDT 

Ao 

AoA 

APB 

ASN, RDA 


Acquisition Decision Memorandum 

Acquisition Category 

Achieved Availability 

Inherent Availability 

Administrative and Logistics Delay Time 

Operational Availability 

Analysis of Alternatives 

Acquisition Program Baseline 

Assistant Secretary of the Navy; Research, Development, and 
Acquisition 


Component Acquisition Executive 
Cost as an Independent Variable 
Concept Exploration 
Commandant of the Marine Corps 
Corrective Maintenance Time 


Defense Acquisition Executive 

Defense Acquisition University 
Defense Federal Acquisition Regulation 
Depot Level Maintenance 

Department of Defense 

Department of Defense Directive 
Defense Planning Guidance 

Defense Systems Management College 
Developmental Test and Evaluation 
Deputy Under Secretary of Defense 


Environmental Stress Screening 


Federal Acquisition Regulation 
Failure Modes, Effects and Criticality Analysis 
Failure Reporting, Analysis, and Corrective Action 


General Accounting Office 


Integrated Logistic Support 

Initial Operational Test and Evaluation 
Integrated Product and Process Development 
Integrated Product Team 

Inspect and Repair Only As Necessary 
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KPP 


LCC 
LRIP 
LS 
LT&E 


M&S 


MARCORS YSCOM 
MARCORLOGBASE 


MIL-HDBK 


MLDT 





PMT 
PM/WSM 
POM 


RAM 
R&D 
RDT&E 


Key Performance Parameter 


Life Cycle Cost 

Low Rate Initial Production 
Logistics Supportability 
Logistics Test and Evaluation 


Modeling and Simulation 

Marine Corps Systems Command 

Marine Corps Logistics Base 

Materiel Command 

Marine Corps Combat Development Command 
Marine Corps Operational Test and Evaluation Activity 
Milestone Decision Authority 

Major Defense Acquisition Program 

Military Handbook 

Military Specification 

Mean Logistics Delay Time 

Mission Need Statement 

Measures of Effectiveness 

Milestone 

Mean Time Between Failure 

Mean Time Between Maintenance 

Mean Time to Repair 


Navy Acquisition Executive 
National Military Strategy 
National Security Strategy 





Operations and Maintenance 
Operations and Support 

Operational Requirements Document 
Office of the Secretary of Defense 
Operating Time 

Operational Test and Evaluation 


Program Executive Officer 

Program Manager 

Program Management Office 

Preventive Maintenance Time 

Program Manager/Weapon System Manager 
Program Objectives Memorandum 


Reliability, Availability, and Maintainability 
Research and Development 
Research, Development, Test, & Evaluation 
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RFP Request for Proposal 


RQT Reliability Qualification Test 

SA Supportability Analysis 

SAE Senior Acquisition Executive 

SAMP Single Acquisition Management Plan 

SCM Supply Chain Management 

SECDEF Secretary of Defense 

SEP Systems Engineering Process 

SLEP Service Life Extension Program 

ST Standby Time 

TAAF Test, Analyze, and Fix 

TEMP Test and Evaluation Master Plan 

TOC Total Ownership Cost 

USD (AT&L) Under Secretary of Defense, Acquisition, Technology, and 
Logistics 

USMC United States Marine Corps 

WBS Work Breakdown Structure 

WIPT Working-Level Integrated Product Team 
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APPENDIX B. PROGRAM MANAGEMENT RELIABILITY 
SURVEY 


New Page 2 Page | of 7 


Naval Postgraduate School it 


Sreatere+, So.) VF ef-e be 


Reliability Survey 
Instructions: 


Howse answer the following questions NLT 16 October 02. Your time and offoet is grastly apprectsted and wil 
hopelully acict im identifying common potential intititors of rdiablity management. Resufts from this survey will 


be presented in aggregate form, not program specific. 


Heaee aeewer all questions 8 date or documentstion ts not available, does mot exiet. of hes not been computed, 
please spacity. 


This survey is being Conducted to support research as part of a Naval Postgraduate School teeasis on relinbility 
evenegement challemges withte the Marine Comps ecquisition process, The results of the thems are intended to directly 
Benalit Program Masagers by identifying cammon inhibitors to rafiabiity managemest, why they ocaur, lessons 
earned, and suggestions for reducing the Ieherent risks. The fonss of the research will be 2 commartson of systems ' 
sellability requirements and predicted rellabélity with actes! achieved reliidity of Nelded «patens. 


The raeaarch le limited to a crom?-aection af matuee omticalypecies Monee induced im the Quarterly Raadineas Reporte 
to Congress (MIA] Tonk, Amphibious Assaeit Family of Vehictes, 5 Tom Track Family of Vehicies, HMMNWY Family of 
Vehiches, Light Acmored Family of Vehicles, (VS Family of Velicles, and the M296 Howitzer). The analysis te limited to 
an seseesment of religb®ty mangement and process Issues and coes not specifically address commodity or 
technology driven eefia titty percitbernes 


1. Product Group: 


2. Program Management Office: 














4. Program/ Weapon System Name (specify variant of the system if applicable): 


© miat 7 Lav 

© M198 © D8, Tractor, Medien 

7 aay © M2406, 7,.62mm Machine Gan 
Cc anny © 5-Ton 

© K-48, LVS Power Unit 0 AM/PRC-129 


© Other (Please Specify) SB 


5S. Current Life Cycle Phase: 
© Phase Tl of old S000 aarkes: How long lees the systore bes in tie field post TOC (in paacs)? [ = of yours 


© Niestone C of new SO00 series: How tong has the system been fe the flekl since post-Full 
Rate Production and Deployment (in years)? ts Bie 


6. What was the intended lite cycle of the weapon system? 


Taig iwwe npenavy Wil/MCAP /awvey aap 1127/2002 
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New Page 2 Page 2 of 7 


7, Rank order what you consider to be the TOP FIVE reliability management problems (1 being the most 
severe problem): 

Traditional test and ewalietion RAM metrics ave not supported by our maintenance dete sources (unadie to 
meke 3 valid comperison between RAM requrements end estimates with actual achieved field data) 


Reliability is mot a Kay Performance Paramater in the GAD 

Missing o¢ poorly written ORD reliability requirements 

Miginterpretaton of reliability related performance specifications 

Contesctor mot designing fer relisbilty sufficiently sbove the requirement 

Acquttition streamlining and/or government downsizing 

Contractor not using best commercial practicas 

Poor relieblity planning ard growth planning (test ton Inte) 

Inadequate or vague polkies end guidance (reed updating) 

Insufficient reliability testing to verity requirements 

Too much peése @ tH felt arstem cheaper (Cost goss outweigh faliailry) 

Too much pressure to field eystern more repidty (Schedule goals cutweigh relisbility) 

Lnrealistic reliability requrements with inadequate rationale 

Need mete qualified personne! in reliability mensgement in PM office 

Not consistentt impeoring reliability after fielding 

0 
Additional Comments: 


4, Who within your organi zotion is primarily responsible for reliability activities for this particular program? 
(Check only one) 


oo00000000009000 


© Program Manager (PM) © Test Team Leader 
© Logistics Management Spacialict (LMS) © Ralebility IPT (iF 90, te Ite formally chartered IPT?) CO Yas Nb 
© Project Team Leader (TL) © Prime Contractor 


© Systems Engineering Tean Leader © No one snecifically 


© Lopietice/Suppertebilty Team Leeder © Other [a 


9, How is the system reliability program and Corresponding Management approach farmally docum ented 
within your program? (Please creck only the primary overriding document) 


© Reletility Program Plan C SAMP (Sings Acquisition Management Pian) 
© Contract SOw © No forms! retisbility menagement plan 

© Other (Please Explain) 
© TENP (Test and Evahiation Manter Plan) — a La 


10. Which activities did/does your program implement to recognize and/or evaluate potential failures and 
causes? (Plesee check ALL that analy) 


D Operation Testing T Relichill ty Qualification Test (RQT) 
T Gavelopmental Testing T Process Feillure Mocs and Effects Analysis (PFMEA) 
T Erwirormentel Stress Screening CESS) T Weibull Analysis 
T Retebilty Modeling (O Physics of Failure (POF) 
itp) www-enps.cavy mil MCAPYaurvey.asp 1L2T/ 2002 


126 


New Page 2 Page 3 of 7 


Failure Reperting, Analysis, and Correcthve Action 
CFRACAS) 


Fault Tree anstrsis (7 Parts Cortrol Frogrem 
TM Probablistic Fellura Ascassmant Cote 


Calibre Modes, Effects and Criticality Analysis (FMECAY 53 pot enow 
or FEA 
T Relisbility Cewsloprment/ Growth Test (AD/GT> 


11, Are you aware of any specific DOD or Marine Corps policy /reguiations regarding weagon system 
reliability management? 

SSS 
rehebility?> 

co Mo 

Oo you feel that existing policy and regulations on reliability provide adequate guidsnce? Please 
explain. 


( Releblity aliocetion 





ADDRESSING RELIABILITY IN THE REQUIREMENTS GENERATION PROCESS 


12, What was the documented reliabdility/ availability requirement? Ln what terms was it messured (i.e, 
MTBE, MTBM, A, MTBSA, MTBOMF, MTBEFF, MTBOMA, MIBMAF, etc)? 


eae eR eee, 


13. Was a reliability requirement identified as a Key Performance Parameter (KPP) in the Operational 
Requirements Document (ORD)? If so, was the requirement in an objective and quentifiable form that 
contractors and the rment could east in? Please explain. 


ld, Was the PMO, as the Material Developer, able to influence Incorporation of reafistic reliability 
requirements as port of the ORO process? C Yes © No © Not Sure 
Additional Comments: 


1S, Was reliability included as a factor in the source selection process? 
Ces (wae it 3 significant discriminate? © ves Chop 
Cie 


Aaational Comments; [TO 


16. How are ORD reliability requirements for your program translated into actual contractual reliability 


requirements? 
© CRD perey ans relative to reliability are restated in SOW/Spec (i.e contract requirement te equal to GRD 


regurement) 
© Addkionsl levels of rebabilty sre appted to the contract (Please briefly describe the process) 


http) www_nps. navy. MCA PYaurvey.esp 12/2002 
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Comprehensive rebabitty requirements are not adequately stated in the contract 
(Other (Explain> 





17, Are there incentives employed in the contract tat are specifically tied to achieving system reliability 
performance requirements? 


© Yes (Explain) [ee 


16. If yes, did these incentives achieve their desired effect? 
CyYes CNo © Too carly to tall 


Additional Commants: [7 | 





DEVELOPMENTAL AND OPERATIONAL TESTING 


19. Did the user, tester, contractor and PM office all agree upon the method (made) to be used to 
determine retiability performance during testi 
Yas (if yes, Where ia this docunented? TEMP?) {— = | 
Cc No 
© Not Cartain 


Additional Comments [ ga 


20. Did your pragram have specific LOTE entrance criteria relative to reli abili 
CS ck TS 


CN 


Additional Gommants I a 


21. Was the amount of time and funding allotted for reliability testing during OT sufficient for youre program’ 
© ves ONo 


Adettionel Commants: 


22. Did the system experience any major reli ability test failures? 
© Yes © No 


Additional Gommants I B 


RELIABILITY REQUIREMENTS VS. CONTRACTOR ESTIMATES VS. ACHIEVED RELIABILITY 





23, To what level was your system ‘'s ORD reliability requirement demonstrated? (state in terns of % of 2D 
requirements met by selecting one from esch column) 


uring DT: During OT During Sustainment: 

© 100% © 100% © 100% 

© +90 % © +90 % © +90 % 

© 780% © +30 % © +80 % 

© +70 % © +70 % © >70% 
http) www_nps.navy milMCAPYaurvey.asp 12/2002 
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© 200% © 210% C 200% 

© 250% © >50% © >S50% 

© 20-50% © 20-50% © 20-50% 

Oo <2% © <2S © <0% 

© DoNOT krow © DoNCT know © DoNOT know 
© Not Applicebe © Not Applicable © Not Applicable 


24. To what level was the contractor's reliability estimate demonstrated? (state in terms of % of estimate met 
by selecting one from each column) 


Curing DT: Quring OT During Sustanment: 
© 100% © 10% ©o10% 

© +90 % oC +90% o 90% 

© +60 % © 780% © +80 % 

© >70% © >70% © 270% 

© 260% © 260% c 2% 

© 250% © 23H © >50% 

© 20-50% © 20-S0% © 20-50% 

oO <me © <2%h Cc <20% 

© DONOT know © DONT know © DoONCT know 
© Hot Applicable © Hot Applicable © Not Applicable 


25, Was the initial contractor reliability estimate documented? 
ves © Nod Cf yes, on what document?) 


Ts such documentation retained? 


© ves © Na (iFyes where (physically) and by whom) [a 


26, What was the initial (prime) contractor reliability estimate for the system? 






Tn what terms was it measured (1.¢. MTBF, MTEM, A)? 


——————————————————————————————s 


27. Has actual achieved reliability of the fielded system been collected/computed? © Yas No 
Lf so, in what form (i.¢., MTBF, MTBM, A,, etc)? 


_ ae: 


26, Whathas been the overall achieved reliability of the fielded system for its entire life cycle thus far? 


29. How Is reliability field data collected to gather failure and repair histories? (Phase check all that sooty) 
f Intermediate SupplyMantenerce Activities MReiability Oats is Net Formalty Collected 


TOepet or 0.5 Maintenance Records Mother expian) 


T Using Unit Readiness Reporting 


30, Does current achieved field reliability deta indicate your system meets, exceeds, or has failed to meet 
the ORD reliability requirement? 


© achieved reliability hes exceaded ORD requirements 
© Achiewed retianility has met ORD requirements 
© achiewed refiability has fallen short of ORD requirements 


http) www-enps.navy milMCAPYaurvey.asp 1L2T/ 2002 
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© Ralisbility data not formally colected 
© Requiremerte were nct delineated in the ORD 


Additional cormimvents: 


31, Does current achieved field reliability dats indicate your system meets, exceeds, or has failed to meet 
the contractor reliability estimate? 

© aAchiewed reliability hes exceeded cortractor agtimates 

© Achivwad reliability has met contractor estimates 

© Achiewad retiability has fallen short of cortracter estimates 

© Rallsbility data mot formally collected 

© Contractor estimates hawe not bean maintained for comparkon 


Acdinonal comments I aj 


32. What specific aeganizatian(s), if any, have compared and assessed actual achieved reRahbility of fielded 
systems to the original contractor estimates? 





If completed, was it done as an official responsibility/role assigned by higher headquarters or was the 


—— conducted far some other reason Le. internet — Please ae 


33, Has the system undergone a Depot Level Maintenance Program (SLEP/PIP /Rebuild/TROAN )? 
C¥e C No (if so, which one aid st whe stage/year in its Ife cycle?) 





34, Following any Depot Level Maintenance Program, has te achieved reliability of the system drastically 
changed? © Yes (No [If yes, please axplain> 


5. Has any of the reliability failure data collected led to identification of O8S cost drivers that subsequentl 
led to cost effective improvements? 


© Yes (Plagse axplain in more detail) I 3 


Cho 


Addtions! comments: Ce 
36, Is there a formed reliability growth for your system? (Le. FRACAS 
© Yes (If yes, where is it documented?) 


CN 


Additions! comments I | 
37, Does your system employ a Reliability Centered Maintenance ran? 
C Yes (If yes, how is it fonnaly inpkementad?) [ | 


Cho 


Adettional comments: [a 








http) www_nps. navy milMCAPYaurvey.asp 1L2T/2002 
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APPENDIX C. MARINE CORPS MATERIEL COMMAND 
MTBF/MAINTENANCE STUDY 


D0209 (MK-48) MEAN TIME BETWEEN FAILURES/MAINTENANCE STUDY 


Assigned: 8 June 2002 

Completed: 

Completed by: Capt Jake Enholm 

Assisted by Jaeyong Lee, Assistant Professor of Statistics, Penn State, Maj Humpert, 
Capt Paige, Capt Frey, CWO-2 Peterson, Deborah Whitley, Mike Everly, Dennis Cooper, 
GySet Pelligrin. 


Objective: Formulate a methodology for determining systemic mean time between 


failures and equipment parts (NSNs) failure rates using MIMMS/ATLASSII/SASSY data 
fields. 


Data Used: D0209 ERO (Equipment Repair Order) Data from 1999 -2001 
Number of Records: 34,592. 


INTRODUCTION 


This study was conducted in order to determine whether mean time between 
failures for a primary end items using current warehoused maintenance management data 
is possible. The study was also conducted to determine whether mean time to failure for 
a particular part on a primary end item using the same data is possible. 

It is important that the study includes the reason why we are attempting to 
calculate mean time between failures for equipment. The calculation of this time is 
necessary in order to determine whether the mean time between failures is increasing, 
decreasing, or remaining constant with age. As equipment ages, its mean time between 
failures decreases until the cost of keeping that item operational is more than the cost of 
buying a new item. Estimates of when maintenance costs will exceed acquisition costs 
are questionable without mean time between failures calculation. Each piece of 
equipment we are concerned with is a system of working parts, and we ref er to its failure 
as a systemic failure. A complete estimate of maintenance costs should concentrate more 
on mean time between maintenance instead of mean time between failure in order to 
capture all costs. 

Mean mileage between failures vice mean time between failures might be a better 
indicator for item replacement in some equipment, such as trucks. It was established in 
an earlier study that the MK-48 meter reading field of the Marine Corps Integrated 
Maintenance Management System (MIMMS) and ATLASS II data that records odometer 
information was unusable for most records (Enholm, 2002). It might be possible to 
construct a sophisticated algorithm to determine a mean mileage for the fleet, but the 
nature of the current study requires lower error rates than are currently found in the meter 
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reading field. It is this reason why mean mileage between failures cannot be accurately 
determined at this time. 

The calculation of mean time to failure for a part in a system is important in order 
to statistically isolate a problem area. Mean time to failure calculation is also important 
when comparing the amount of time we can expect a part to be in operation, compared to 
its cost. 

In the calculations used in this study, mean time between failure (MTBF) and 
mean time between maintenance (MTBM) periods were combined. The addition of the 
deadline control date field can be used to tell if a vehicle is deadlined or just operating in 
a degraded fashion. The deadline control date field is left blank on equipment repair 
orders (EROs) that are not deadlined. This study substitutes median time between 
failures/maintenance for mean time when discussing alternate methodologies. ! 


SYSTEMIC MEAN TIME BETWEEN FAILURE/MAINTENANCE 


The calculation of a time to failure/maintenance of an item requires that we know 
the time an item has been in operation. For systemic failure or maintenance degradation 
there is also a requirement that the age of the system be established. This is particularly 
important when attempting to establish the age when an item starts spending longer and 
longer times down as it gets older, and maintenance starts adversely affecting operations. 

Unfortunately, there is not a central repository or data warehouse where the 
Marine Corps has information establishing the age of a serialized item. The program 
managers of an item keep a logbook of serial numbers that detail when a serial number 
was fielded. This information is not always in electronic format, or readily available. 
For this particular study, information fom MARCORSYSCOM helped establish the 
number of MK-48s that were fielded between the years 1985 and 1999. 


Table 1. MK-48 Fielding Numbers by Year. 





These numbers were used to estimate the year a serial number was fielded. 
Estimation had to be done in this case because no database was available that specifically 
contained this information. 

Serial numbers are given sequentially as an item is fielded. Using the number of 
MK-48s fielded each year combined with the serial numbers found in the ERO header 


1 Statistically, this is not the same number. The sample median time between failures is a point in time 
where 50% of the samples fail. The sample mean time is the average time between failures. Because we 
cannot always calculate the mean reliably when using certain statistical methods, we will sometimes have 
to substitute a median as the most reliable substitute. 
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records, a table was constructed that estimated the ages of the serial numbers from 1985 - 
1989. 


Serial Yr 
Number | Fielded 


Table 2. A Portion of the MK-48 Serial Numbers — Year Fielded Table. 





These were estimated by sorting the serial numbers found in maintenance 
management data and counting them out according to the totals in Table 1. 

The maintenance cycle of an item such as the MK-48 is commonly referred to as 
a repairable process because the item transitions between operating and non-operating 
status. The model used here is a maintenance model, where both preventive and 
corrective maintenance actions are applied to systems being studied. Leemis (1997) 
advocates the use of a non-homogeneous Poisson process to model failures for repairable 
systems since it can model deteriorating systems. Non-homogeneous means that the time 
between failures can increase or decrease. For a definition of a Poisson process, see 
Appendix A. 

Once the age of a vehicle has been established, the time between failures can be 
correlated with the age of the vehicle if a wide enough data range is established. 
Unfortunately, although the Marine Corps has maintenance management data 
warehoused back to 1998, the data from 1998 appears to be incomplete. There were 
between 8000-13000 equipment repair order (ERO) records recorded for MK-48s from 
1999-2001. But in 1998 there were only 1400 records, suggesting that the warehousing 
was incomplete and the 1998 data should not be used. This assumption reduced our 
available data range to 1999-2001. 

Systemic failures were compiled using the assumption that a group of EROs with 
a common serial number and the same date received in shop (DRIS) entry consists of one 
job for a particular vehicle. The time between DRIS and the date the job was closed (all 
the EROs for that job are closed out) is also of particular value. The number of days the 
vehicle was worked on for a particular job reduced the amount of time that the vehicle 
was available. In general, if two EROs have overlapping dates, the dates that cover the 
largest period of time are used for failure calculation. 


CENSORED AND UNCENSORED DATA 


The use of data from 1998-2002 involves the use of censored data. Censored data 
involves the use of incomplete amounts of time in some samples because our records 
stopped at a particular date. If we have a serialized piece of equipment, such as MK -48 
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515814, and we know when in 2001 it was worked on, but we didn’t see it worked on 
again before we ended our study in 2002, then we say our data is right censored for that 
period of time. If we have another serial number, 515842, and we know it was worked 
on once in 2001, and again in 2002, then we know the amount of time that serial number 
was running before it was worked on again. We say that that sample was uncensored. 
Both censored and uncensored observations provide clues to generating a mean or 
median time between failure/maintenance. The statistical calculations were done using 
Kaplan-Meier survival estimates, which estimate the probability that an item will survive 
to a particular time by conditioning on the probability that it survived up to the previous 
period. For more information, see Kaplan and Meier, 1958. 

Kaplan-Meier survival statistics were a useful tool to describe the MK-48’s 
failure/maintenance periods. Figure 1 shows the Kaplan-Meier (K-M) survival 
probabilities for operating a MK-48 up to a particular day based upon maintenance 
management system data from 1999. Data calculation for a median for 1999 data was 
attempted, to see if a distinct median for every year of warehoused data could be isolated. 
If so, we could see if the medians for several years were non-homogenous. The gaps on 
the right part of the graph show the censored data observations. A median time to failure 
could not be calculated because there were too many censored observations. 


K-M Survival Probability and Gaps for 
1986 MK-48s in 1999 
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Days before Failure/Maintenance 





Figure 1. Kaplan-Meier survival probabilities show ing the probability that a MK-48 
fielded in 1986 will operate up to a certain day before needing maintenance. The data 
used was from 1999 maintenance EROs. Data that was censored, or incomplete shows 
up as gaps. The large amount of censored data did not allow a median to be calculated. 


Because a median could not be calculated using just the 1999 data, the data period 
was widened to include 1999-2001. Table 3 shows the results of that analysis. 
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Percentiles of (all years) 





Table 3. Analysis output from the software program Statistica showing Kaplan- 
Meier survival statistics for MK-48s, using maintenance management data from 1999- 
2001. The statistics reflect a median time to failure/maintenance of 181 days. 


The median time between failure/maintenance of MK-48s of 181 days seemed 
excessive given current readiness rates, however it must be taken into account that this 
rate includes all MK-48s in the Marine Corps. The Mk-48s in stores, and on Maritime 
Pre-positioning Force (MPF) ships might be affecting this trend. Acquisition of an 
additional data set that filters out stores and MPF ship MK-48s is required to test this 
hypothesis. 

It was also possible to calculate the median time between failures/maintenance for 
a group of MK-48s fielded in a particular year. The expected median times between 
failure/maintenance should be decreasing with age. Figure 2 shows that this is indeed the 
case, with the exception of MK-48s that came out in 1987. (This excludes vehicles 
fielded in 1989 because the low number of vehicles fielded in that year) 


MK-48 Median K-M Survival Times 














1986 1987 
Year Vehicle Fielded 





Figure 2. Median K-M times between failure/maintenance correlated w ith estimated 
age. The median times decrease with age, with the exception of vehicles fielded in 1987. 
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Year of MK-48 and MTBM 
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Figure 3. Mean times between failure/maintenance correlated with estimated age 
using the average observed times and discarding the censored observations. The Kaplan- 
Meier method was not used here. The mean times do not decrease with age, but the 
values are more reasonable given current MK-48 readiness rates. 


An alternative method to the K-M methodology was also used. This methodology 
discarded the censored observations and used only samples that contained two jobs or 
more. The amount of time the vehicle was operational between jobs for a similar serial 
number was calculated. This time was then averaged for all observations. The sample 
mean for this method was 41.2 days. The mean seems more reasonable with current 
readiness levels for the fleet, but this methodology did not show increasing MTBF/M for 
increasing age. 

A possible validation of tying in a set of EROs with a similar serial number and 
the same DRIS date as one job was seen. The jobs were correlated with the amount of 
time each vehicle was worked on. An average readiness for the fleet was calculated 
using the amount of time worked on for each job. This calculation was adjusted for a 
data period that corresponded with archived MK-48 readiness in the Material Readiness 
Assessment Module (MRAM). The job-methodology calculated readiness was 82%. 
The MRAM readiness for the same period of time was also 82%. It should be noted that 
the MRAM only uses deadlined vehicles, and the data used in this study was for all 
maintenance tasks. Further calculation with another primary end item is necessary before 
any conclusions can be made on this validation process. 


NSN TIME TO FAILURE 


The equipment repair order records were “drilled down” or linked to National 
Stock Numbers (NSNs) that were ordered for a particular ERO. This would not have 
been possible to compile in a short period of time without the help of the integrated 
SCOPE database construc ted by Capt Paige’s team. The necessary data was compiled 
with the integrated system in three minutes. A parallel effort using the conventional 
maintenance management record data system in place required two weeks. 
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Calculation of a mean time to failure of an NSN was done under the assumption 
that the NSN failed and another was ordered and replaced the failed item. This did not 
include the repair cycle used in secondary repairable items (secreps) such as engines and 
transmissions. Calculation of a mean time to failure for an NSN is problematic because 
there might be several NSNs that are in use for a particular system that perform the same 
function. 

A common statistical method for modeling lifetimes of equipment parts is the 
Weibull distribution, which is a “...generalization of the exponential distribution...” 
(Leemis, p. 88). The Weibull distribution was selected for modeling two of the most 
common NSNs found in the MK-48 maintenance records. Table 4 is a partial printout 
from the statistics software program Statistica and shows parameter estimates that 
Statistica came up with after looking at the data from the selected NSN. The low p- 
values (also known as observed significance levels) on the right indicate that the data 
does not fit the Weibull distribution well. The NSN chosen was one of the most 
frequently encountered ones in the data set. Figure 4 shows the survival probability 
distribution for these samples. 


[Parameter Estimates, Model: Weibull (nsn1) P [| 
ights: 1=1., 2=1,/V, 3=N()*H@ ee ee ee 
PT CWWatriance tare. | =| *i| ——sd 


____ftambda_ambda__fambda_hi-Sqr. fp ____ 
Weight! |__ 0.000664 3.2E-07]__— 0.000565] 18.33028| 9] 0.031563 
Weight2 | ___0.000747|__—2.09E-07] 0.000457 19.10768| 9] 0.024329 
Weight3 | o.00191gf2.01E-06f 0.001417] 19.23488[ 9] 0.023297 


Table 4. Statistica parameter estimates for a MK-48 Weibull lifetime distribution. 
The low p-values indicate a low level of confidence that the NSN distribution fits a 
Weibull curve. 
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Figure 4. Weibull probability of NSN 2510012331768 surviving up to a particular 
day. 
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A second MK-48 NSN that was frequently ordered was analyzed. The high p- 
values that Statistica generated indicate that the data was close to a Weibull distribution 
with the chosen parameters. Statistica gave this NSN a median life time of 1118 days. 
Figure 5 shows the survival probabilities for the parameters in Table 5. 


Parameter Estimates, Model: Weibull (nsn2)_ | | TT 
ights: 1=1,,2=1/V,3=NU)*H) | | TT 
| Wariance[Std.Err. | Wariance | || 


| 

| Lambda _Lambda___|Lambda_Gamma_|Gamma__[Chi-Sqr. laf [p _| 
Weight 1_| 0.000791] _ 1.07E-06 |0.001035 | 0.864681] 0.040519 6.143479] 9] 0.725468 
Weight 2_| 0.001402] _ 1.37E-06 |0.001171 | 0.789683] 0.015711] 5.995469] _9|_ 0.740362 
Weight 3 | 0.001913] _ 2.82E-06 [0.001679 | 0.728864] 0.017097] 6.031236] 9] _ 0.736779 


Table 5. Statistica parameter estimates for a MK-48 Weibull lifetime distribution. 
The high p-values indicate a high level of confidence that this NSN distribution fits a 
Weibull curve. 





MK-48 NSN 2540012348073 Cummulative 
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Figure 5. Weibull probability of NSN 2540012348073 surviving to a particular day. 
The median (not the mean) number of days for survival was 1118 days. This number 
indicates the curve’s downward trend is steeper after 1118 days. 


CONCLUSIONS AND RECOMMENDATIONS 


The mean-time between failure/maintenance analysis of the MK-48 revealed 
some areas where maintenance management data systems can be improved. The 
improvements will make this type of analysis more accurate, and more useful with 
respect to estimating when a system’s mean time between failures/maintenance is 
increasing or decreasing with age. The study produced two different methodologies to 
calculate time between systemic maintenance/failure. The analysis of MK-48 NSNs and 
their median time to failure also revealed some areas that can be beneficial to producing 
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useful results. 
The following recommendations would increase the accuracy of systemic and NSN 
failure rate calculation: 


1) A database containing primary end item serial numbers, the year fielded, and their 
cost needs to be developed. A second database where the NSNs of a primary end 
item, their description, and their cost needs to be constructed. NSNs that perform 
the same function need to be cross -referenced. 


2) The current warehoused maintenance data of the Marine Corps needs to be 
extended back in time as far as possible. 


3) Perhaps the best alternative for error checking of serial numbers will be provided 
with the implementation of the Global Combat Support System -Marine Corps 
(GCSS-MC). The GCSS-MC system will replace our current legacy maintenance 
systems and could contain serialized records for each primary end item in the 
Marine Corps. If a manpower -type record of information for a serial number can 
be checked when a new ERO is entered, a calculation can be done to see if the 
new entry makes sense. A sophisticated algorithm known as an intelligent-agent 
can run through a series of decision trees that look at past dates and entries for 
meter readings for that serial number. The intelligent agent then makes a decision 
whether or not a meter reading is reasonable for that serial number. If not, a 
notification back to the Maintenance Management Officer of the unit that made 
the entry can be sent with a request for clarification of the new entry. 


It is possible to calculate a sample mean or median time between 
failures/maintenance for some of the equipment in the Marine Corps using the 
methodologies presented in this study. The accuracy of such analysis will be suspect if 
the current weaknesses of the system are not fixed. A validation of the most accurate 
method is currently being conducted. 


APPENDIX A 


Ross defines a Poisson process as a “The counting process {N(t), 0} is said to 
be a Poisson process having rate ?, ?>0, if 


i N(O)=0 
ii. The process has independent increments 


ii. The number of events in any interval of length ¢ is Poisson distributed with 
mean 7. That is, for all s, t=0 


P{N(t+s)—-N(s)=n}=e™ My, n=0,1..." 
Nn: 
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Notes on Julian Dates 


A challenge encountered was the way dates are entered into EROs. At the time 
MIMMS was implemented there were many reasons why a Julian dating system was used 
for date entries. In the year 2002, the system is a hindrance and not necessary. The way 
the system works now, the mechanic takes a standard date and with the use of a Julian 
date calendar converts the date and inputs it into the system. Then the individual looking 
at the data looks at the Julian date and converts it back into a standard format that is 
understandable. 

The Julian date format is also problematic when using it in Excel or Access, the 
two most common forms of data manipulation software. Excel can calculate the number 
of days between two dates by simply subtracting the two date in standard month/day/year 
format. Excel automatically does the rest. When the date is in Julian format, string 
extraction functions must be used that convert the field into standard month/day/year and 
then the calculation can be performed. The strings that the Julian dates are stored in are 
also problematic. Excel has a problem properly sorting these strings. 


MTBE/M FORMULATION 

Indices 

J job number 

y year of job 

s serial number of equipment 

Variables 

Giy,s The date that an ERO or group of EROs in a job was opened. The 


date the group of EROs was opened should be the earliest date 
received in shop (DRIS) in the group. Job 7 in of equipment in year 
y with serial number s. 


Oiy,s The date that an ERO or group of EROs in a job was closed. The 
date the group of EROs was closed should be the latest date in the 
group. Job j in of equipment in year y with serial number s. 


Dj-y,5 The number of days between jobs j andj + J. 

Cay, s Censored time for job j, in year y, for serial number s. This is the 
time from the date the job was closed to the end of the data taking 
period. 

N, Total Number of Jobs for a serial number. 
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L Total number of serial numbers in equipment being analyzed. 





5 Censoring indicator: 1= censored, 0 = uncensored. 

T, Survival time observation for serial number s. 

Formulation 

Djry,5= Gjsty,s- Ojy,s Calculation of the number of days between jobs 

X, = yy . Calculation of mean number of days per job for a 
i 1 particular serial number. 

T =min(X,C) The minimum value between X and C is picked for 

T. 
& =l(X<C) If X is less than Cthen 6 =0,else 5 = 1. 


Two methods were used for comparison of systemic mean time between 


failures/maintenance in this study: 


1) A, = Kaplan-Meier (K-M) product limit estimate for T =min(X,C) and 6 = 


I(X<C) 


2) 2, = 1/X, = The inverse of the average days between jobs for the observed 


samples. This methodology discards the censored observations. 


1) 


2) 


1/1, 


6 Mean Time Between Maintenance/Failures for equipment with age 


a. (Method 2 only) 


Notes from calculating systemic Median Times Between Failures/Maintenance: 


Averaging the number of days between jobs on the same serial number does not 
calculate points from censored observations that might result from the date that 
the last job was closed until the end of the data period. These points are not taken 
into account. 


The survival data generated with a Kaplan-Meier distribution implies that the 
missing observations need to be extended by increasing the warehousing of data 
back in time in order to get this data. A median can be calculated from the current 
data set, but it would be better to get a median from each year in the data set. 
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APPENDIX B 
Notes from calculating part (NSN) failures: 


1) NSNs were calculated by separation from the main data set and sorted by serial 
number of vehicle they were mounted on. 


2) If an NSN was mounted on the same vehicle two times or more the number of 
days it lasted before another NSN with the same number was mounted on it was 
used as an observation. 


3) If an NSN was mounted on a vehicle and left there until the data period ended we 
count the number of days between the date the job was closed and the date the last 
piece of data was collected. This data is annotated as being right censored. 


4) What if the part wasn’t the same one that was mounted before? For example we 
replaced the right headlight, then the left headlight goes out and we replace that? 
We have no way of telling. 

5) What if the NSN was replaced by a different NSN for some reason? We don’t 
know this, but it can be fixed with an NSN database. 
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